Your description here initially made me think you cannot connect at all, but when the link describes a grey screen with an X cursor... that's something I've come across a few times.
There is a script executed by XRDP on successful login which defines how to start the desktop environment. Presumably because the server install doesn't include any desktop (or an X server) by default, this will be blank or points to the wrong thing.
The file you want is /etc/xrdp/startwm.sh
I've written a forum post a few years ago for Lubuntu (LXDE), but a similar thing can be applied for the MATE desktop. Instead of lxsession, it's mate-session.
In a nutshell, a compact 1-2-3 fix:
- Open the script (which is executed on successful logins):
sudo nano /etc/xrdp/startwm.sh
- Comment out by putting a hash (#) in front of:
. /etc/X11/Xsession
- Add at the bottom:
mate-session
Then press CTRL+X and y to save.
Changes should take effect immediately for the next xrdp session. If not:
sudo service xrdp restart
(This will kill any existing xrdp sessions that are still running)
Just a note: If you decide to switch away from MATE to a different environment for remote desktops on the server, you'll need to change mate-session to the corresponding desktop.