[WIP] xrdp on macOS (with ulalaca)

Disclaimer

xrdp on macOS is not officially supported by the xrdp team so far, but we welcome your contributions and volunteers. This document is brought by contributors. Feel free to edit this document but we xrdp team are not involved xrdp on macOS.

To whom want to use xrdp on macOS, SHARE YOUR KNOWLEDGE WITH EACH OTHER.

There's also a gitter chatroom to have talk with xrdp on Mac users.

• https://gitter.im/neutrinolabs/xrdp-mac

FIXME: This guide assumes that your Mac is Intel x86_64.

NOTE: librfxcodec has not optimized for Apple M1 (arm64) yet. You can contribute to librfxcodec by porting x86_64 assembly code to arm64.

Tested Versions

architecture	macOS	xrdp	ulalaca	ulalaca-xrdp	note
x86_64 (Hackintosh)	12.3.1	dee2992 unstabler/macos_h264enc	a65f739	f06a166	VideoToolbox support available, but RFX has broken VideoToolbox API requires fork=false in /etc/xrdp/xrdp.ini xrdp crashes on session disconnection
x86_64 (Hackintosh)	12.3.1	0.9.19	a65f739	f06a166	RFX takes too much bandwidth (~10MB/s) when playing videos.

Tested Clients

name	platform	host arch	xrdp/ulalaca	working	note
mstsc.exe	Windows 10	x86_64	0.9.19; ?	❓	tested with RFX ([email protected]); disconnected randomly with protocol error
freerdp (2.7.0; X11)	Linux	x86_64	0.9.19; ?	⭕	tested with RFX ([email protected])
mstsc.exe	Windows 10	x86_64	dee2992; implement-resize	⭕	tested with GFX(AVC/yuv420)
freerdp (2.7.0; X11)	Linux	x86_64	dee2992; implement-resize	⭕	works best with /gfx:AVC420 /u: +toggle-fullscreen /smart-sizing
Microsoft Remote Desktop	Android	x86_64	dee2992; implement-resize	❌	client requires rfx codec but broken (built with [email protected])
Microsoft Remote Desktop	iOS	x86_64	dee2992; implement-resize	❌	client requires rfx codec but broken (built with [email protected])

Building xrdp on macOS

Prerequisites

• Install Xcode (or Xcode command line tools: xcode-select --install)
• Install Homebrew: http://brew.sh/
• Install OpenSSL from Homebrew: brew install openssl
• Install Automake + Autoconf: brew install automake
• Install Libtool: brew install libtool
• Install pkgconfig: brew install pkgconfig
• Install nasm: brew install nasm (needed by librfxcodec)

Build xrdp

TODO: describe about install xrdp as daemon

With VideoToolbox support

Click to expand

NOTE: VideoToolbox provides hardware-accelerated H.264 encoder.

FIXME: since PR #2263 has not merged yet, you should clone unstabler/xrdp instead of upstream.

1. Get the copy of xrdp source code.

$ git clone https://github.com/unstabler/xrdp.git xrdp
$ cd xrdp
$ git checkout dee2992

2. Fetch source code of ulalaca-xrdp into xrdp.

$ git clone https://github.com/unstabler/ulalaca-xrdp.git ulalaca

3. Apply patches.

FIXME: save below diff text as 'remove-xorgxrdp_helper.patch'.

diff --git a/Makefile.am b/Makefile.am
index dddc5d0c..88ee9504 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -59,8 +59,7 @@ SUBDIRS = \
   pkgconfig \
   $(XRDPVRDIR) \
   tests \
-  tools \
-  xorgxrdp_helper
+  tools

 distclean-local:
 	-rm -f xrdp_configure_options.h
diff --git a/configure.ac b/configure.ac
index 3465a960..81039c2b 100644
--- a/configure.ac
+++ b/configure.ac
@@ -562,7 +562,6 @@ AC_CONFIG_FILES([
   xrdp/Makefile
   xrdpvr/Makefile
   xup/Makefile
-  xorgxrdp_helper/Makefile
 ])

 AC_REQUIRE_AUX_FILE([tap-driver.sh])

# remove 'xorgxrdp_helper' from build targets
$ patch -p1 < remove-xorgxrdp_helper.patch

# Add ulalaca as build target
$ patch -p1 < ulalaca/xrdp-automake.patch

# force use BGRA pixel format in RFX encoder
$ patch -p1 < ulalaca/xrdp-encoder-force-use-bgra.patch

4. Build and install xrdp.

NOTE: These commands will install xrdp into /opt/xrdp.

./bootstrap
./configure --enable-pixman --enable-videotoolbox --enable-ulalaca PKG_CONFIG_PATH=/usr/local/opt/openssl/lib/pkgconfig --prefix=/opt/xrdp
make -j
sudo make install

Without VideoToolbox support (0.9.19)

Click to expand

1. Get the copy of xrdp source code.

$ curl -L https://github.com/neutrinolabs/xrdp/releases/download/v0.9.19/xrdp-0.9.19.tar.gz > xrdp-0.9.19.tar.gz
$ tar xvzf xrdp-0.9.19.tar.gz
$ cd xrdp-0.9.19

2. Fetch source code of ulalaca-xrdp into xrdp.

$ git clone https://github.com/unstabler/ulalaca-xrdp.git ulalaca

3. Apply patches.

# Add ulalaca as build target
$ patch -p1 < ulalaca/xrdp-automake.patch

# force use BGRA pixel format in RFX encoder
$ patch -p1 < ulalaca/xrdp-encoder-force-use-bgra.patch

4. Build and install xrdp.

NOTE: These commands will install xrdp into /opt/xrdp.

./bootstrap
./configure --enable-pixman --enable-ulalaca PKG_CONFIG_PATH=/usr/local/opt/openssl/lib/pkgconfig --prefix=/opt/xrdp
make -j8
sudo make install

Edit configuration file

Click to expand

1. Open /etc/xrdp/xrdp.ini, and add below contents.

[Ulalaca]
name=Ulalaca
lib=libulalaca.dylib
username=ask
password=ask
#pamsessionmng=127.0.0.1

2. Restart xrdp.

Build Ulalaca

Click to expand

1. Get the copy of Ulalaca source code.

$ git clone https://github.com/unstabler/ulalaca.git

2. Build sessionbroker

NOTE: sessionbroker authenticates the user login, and assigns appropriate session.

$ xcodebuild DSTROOT=/usr/local/opt -workspace Ulalaca.xcworkspace -scheme sessionbroker
$ sudo xcodebuild DSTROOT=/usr/local/opt -workspace Ulalaca.xcworkspace -scheme sessionbroker install

3. Register sessionbroker as Service (LaunchDaemon)

$ sudo cp /usr/local/opt/ulalaca-sessionprojector/pl.unstabler.ulalaca.sessionbroker.plist /Library/LaunchDaemons
$ launchctl load -w /Library/LaunchDaemons/pl.unstabler.ulalaca.sessionbroker.plist

4. Build sessionprojector.app

NOTE: sessionprojector.app relays the contents of the user session screen to xrdp, and posts keyboard, mouse events received from xrdp.

# build sessionprojector
$ xcodebuild DSTROOT=`pwd`/build -workspace Ulalaca.workspace -scheme sessionprojector install

# copy sessionprojector.app /Library/PrivilegedHelperTools
$ sudo cp -rv build/Applications/sessionprojector.app /Library/PrivilegedHelperTools

5. (Optional) Register sessionprojector.app as LaunchAgent (for auto-start on each users login session)

$ sudo cp sessionprojector/LaunchAgents/pl.unstabler.ulalaca.sessionprojector.plist /Library/LaunchAgents/

6. Grant permissions to sessionproject.app

7. Reboot and check if the Ulalaca icon appears in the menu bar.

Troubleshooting

General

• Since surface resizing is not implemented yet, an error may occur if the client's resolution is different from the host's screen resolution.