You are trying to install xrdp on CentOS 6.3 running on ESXI 5 with vmware installed (running without vmware tools shouldn’t be a problem with the vnc server). I took me a lot a time to figure out, but is actually works.
Personally I’m running with my firewall down on this setup, but you should be able to open 3389 if that makes you fell better.
Install xrdp (http://opensourcebyte.blogspot.dk/2011/01/installing-xrdp-on-centos.html)
1) Add EPEL Repo
a. Download EPEL RPM
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-7.noarch.rpm
b. Import KEY
wget http://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-6
rpm --import RPM-GPG-KEY-EPEL-6
c. Install EPEL
rpm -i epel-release-6-7.noarch.rpm
2) Install xrdp
yum install xrdp
3) Start xrdp
/etc/init.d/xrdp start
chkconfig xrdp on
You should be able to connect to the Linux but, but you end up with a black screen terminal and unable to use the mouse or keyboard. You should also get something like this: Authentication is required to set the network proxy used for downloading packages. Or something similar.
Some very cleaver guy on vmware community suggested this (http://communities.vmware.com/thread/326868):
Rename the /usr/bin/vmware-user executeable, to stop execution on X-Windows startup.
With the solution you don't have to uninstall the whole vmware tools package and will only lose the mouse pointer/screen resulotion/cut'n'paste ... things when using the VMware vSphere Console Application. No impacts on VNC-Session, except that this is now functional.
It is not very nice workaround too renaming files, and a better man than me would be able to disable the auto start mechanism - but somehow I haven't made that happen.
Restart and you should be able to connect properly! The End.