Skip to content

Latest commit

 

History

History
256 lines (194 loc) · 11.3 KB

README.md

File metadata and controls

256 lines (194 loc) · 11.3 KB
Raw

These instructions are based on the notes from the NA-MIC Project week 31 - GCP setup for Slicer project. The original notes were copied here for more convenient maintenance and refinement.

Slicer in Google Cloud Platform (GCP) with GPU support

Objective

Replicate Slicer running GCP machine with instructions and write them down for the public

PLEASE READ: Important notes

  • The Google Cloud Platform costs real money once your free trial is over. Be sure to shut down anything you aren't using or your credit card will eventually be charged.
  • Be careful with your login information. If someone takes over your account they could run up a huge bill that you will be responsible for paying.
  • Depending on your specific situation, not all of the options or steps may be available or applicable (e.g., as a member of organization, your organization administrator may have disabled some of the options discussed).
  • Unless you are not concerned about billing, remember to SHUT DOWN THE MACHINE when you aren't using it! You are billed continuously while the VM instance is running.
  • Even after you stop the VMs, you keep paying for the disk storage attached to those machines! You can delete the VM instances to stop incurring those costs.

Instructions

  1. Sign up for 300$ free credit on GCP
  2. Go to https://console.cloud.google.com/home
  3. Select left sidebar "Compute Engine --> VM instances"
  4. Create Instance with the configuration as you wish
  5. Machine type —> Customize and select GPU
  6. Select “Ubuntu 18.04” or "Ubuntu 20.04" boot disk
  7. Increase the size of the boot disk from the default 10Gb to a larger number (consider 100Gb).
  8. Finish creation of VM
  9. (Optional) Create instance templates for repeated creation

Important: GPU usage prerequisites**

  1. Go to sidebar —> IAM & admin —> Quotas
  2. Select metrics —> None and search for GPU
  3. Select GPUs (all regions)
  4. Check and click EDIT Quotas
  5. Enter your information and click next
  6. Set limit to 2
  7. Request process might take up to 2 business days, but if you send them an email, they could be faster with it (at least for me it was)

You have two options to access VNC:

  • Option 1: by directly connecting to the noVNC port on the GCP VM instance - you will need to adjust your firewall settings. Important: this approach is not secure - it will allow anyone to connect to the VM!
  • Option 2: by tunneling the connection through an SSH channel - this approach is easier to implement, and will restrict access to the VM instance to authorized users only

Option 1: Direct access to noVNC port on VM instance (not secure!)

Configure Firewall to open the noVNC port:

  1. Under VM instance configuration: Firewall --> allow HTTP and HTTPS
  2. Select left sidebar "VPC network --> Firewall rules"
  3. Select "CREATE FIREWALL RULE"
  4. Set Source ip ranges: 0.0.0.0/0
  5. Protocols and ports: tcp: 6080

Option 2: Access noVNC port via SSH tunnel

Configure prerequisites on your machine:

  1. Install gcloud SDK as described here.
  2. Set up gcloud with your GCP credentials:
$ gcloud init

Configure the VM instance

  1. Start VM by clicking
  2. Get terminal access to the VM instance by either clicking on "SSH" button in the VM instances list, or executing the following command in the local terminal on your computer (considering you installed the GCP SDK, as discussed above):
$ gcloud compute ssh <VM instance name>
  1. Install the prerequisites (takes a few minutes)
sudo apt-get -y update && \
  sudo apt install -y ubuntu-drivers-common && \
  sudo ubuntu-drivers autoinstall && \
  sudo apt install -y xinit && \
  sudo apt-get install -y x11vnc && \
  sudo apt-get install -y xterm && \
  sudo apt-get install -y libpulse-dev libnss3 libglu1-mesa && \
  sudo nvidia-xconfig && \
  sudo apt-get install -y python && \
  git clone https://github.com/novnc/noVNC

Execute the following and take note of the BusID

sudo nvidia-xconfig --query-gpu-info

Open the X11 configuration file

sudo vim /etc/X11/xorg.conf

and insert the following BusID line using the BusID value you retrieved earlier into this Section:

Section "Device"
   Identifier     "Device0"
   Driver         "nvidia"
   VendorName     "NVIDIA Corporation"
   BusID          "PCI:0:4:0"
EndSection

or if /etc/X11/xorg.conf does not exist, create a file in /usr/share/X11/xorg.conf.d/xorg.conf with the contents listed above.

To set the size of the virtal display you can edit the Screen section as shown below.

Section "Screen"
    Identifier     "Screen0"
    Device         "Device0"
    DefaultDepth    24
    Option         "UseDisplayDevice" "none"
    SubSection     "Display"
        Virtual     1920 1080
        Depth       24
    EndSubSection
EndSection

Note: if the Slicer font and icons size is too small, run xrandr to find the max resultion and use that as the Virtual setting. Then use xrandr or install axrandr to resize the desktop to a comfortable size.

On some devices (e.g. nvidia A100) xinit will fail if you have the Option "UseDisplayDevice" "none" option, so delete that line for that device.

Server-side: Start X11, VNC and noVNC

Each reboot (e.g. after doing 'start' on the google cloud console). The commands below are set up so you can cut and paste them into the ssh terminal from the google interface, but if you want to debug more easily them you might want to paste each in its own terminal.

sudo xinit -- +extension GLX &
./noVNC/utils/launch.sh --vnc localhost:5900 &
while true; do x11vnc -forever -display :0; sleep 1; done

Note that the while true ... part in the instructions above is needed to address the possible intermittent crashes of x11vnc. You can improve stability by building x11vnc from source (see Troubleshooting section).

Server-side: Download and unpack Slicer for linux

Here using a specific revision, but any version should work

wget http://slicer.kitware.com/midas3/download/item/435293/Slicer-4.10.2-linux-amd64.tar.gz
tar xvzf Slicer-4.10.2-linux-amd64.tar.gz

Client-side: Connect VNC

Note: this is a very raw linux machine and you are running as root. There is also a user account under your name that is automatically created by the google VM boot process. Pretty much anything from the last few decades of linux development should run the same here as it does on a local workstation.

Option 1: Direct access to noVNC port on VM instance

  1. Connect to http://{VM_External_IP}:6080/vnc.html
cd Slicer-4.10.2-linux-amd64
./Slicer

It is better to start Slicer as a non-root, since otherwise it will not be possible to install extensions. You can do this with

$ su <user name> && ./Slicer

Option 2: Access noVNC port via SSH tunnel

  1. Tunnel noVNC port:
 gcloud compute ssh <your VM name> --project <your GCP project name> --zone <your VM zone> -- -L 6080:localhost:6080
  1. Open the connection in your browser: http://localhost:6080/vnc.html

TODO

If anyone works on these issues please write them up and let us know:

  • Add instructions for setting up TLS / HTTP (e.g. with letsencrypt.org
  • Add instructions for setting up reverse proxy with OAuth
  • Add a window manager and other utilities to the X environment, (e.g. with OpenBox, as is done here)
    • running sudo apt-get install -y openbox && openbox-session in the terminal window is one way to start. A lot of things won't work out of the box but you can configure the files in /etc/xdg/openbox. (E.g. edit /etc/xdg/openbox/menu.xml and change the <execute> section to xterm.
    • Also you can access the NVidia X server settings to change the screen resolution.
  • Describe other VNC options
  • Come up with similar instructions for AWS and Azure (and other computer rental providers).
  • Consider setting up a multi-user system with multiple logins to be used in 'time sharing' for collaboration and resource sharing.
  • Explore reliability (sometime instances reboot unexpectedly)
  • Explore the most cost-effective options (e.g. preemptable instances and GCP vs AWS and others for running tasks)

Troubleshooting

Sporadic x11 server disconnects

If you have trouble with the x11 server disconnecting when openning menus or resizing files, you are probably hitting this bug which is not yet fixed in ubuntu.

You can replace with a patched version like this (as root):

curl "https://drive.google.com/uc?id=1FCTxYPAPf58AqchST0SLYfZFZoVANCfL&export=download" -o x11vnc -L
sudo cp x11vnc /usr/bin/x11vnc
  • For key repeat: sudo apt-get install x11-xserver-utils and then xset r rate 300 10 (you also need run xset r on twice to override the -norepeat option of x11vnc)

You can also build x11vnc from source using the instructions below.

  1. Install prerequisites (as discussed here):
sudo cp /etc/apt/sources.list /etc/apt/sources.list~
sudo sed -Ei 's/^# deb-src /deb-src /' /etc/apt/sources.list
sudo apt-get update
  1. Install x11vnc build dependencies, checkout source, configure and build (as in the x11vnc build instructions).
sudo apt-get build-dep x11vnc
git clone https://github.com/LibVNC/x11vnc.git
cd x11vnc
autoreconf -fiv
make

The binary will be in the src directory!

Alternative VNC servers

Some recipes, such as this one, recommend using vncserver, which is a wrapper around xvnc. Based on this post on NVIDIA forum, xvnc does not support GLX, and will not work with Slicer.

If you experiment with alternative VNC implementations, please share your experience via PR!

Debugging

If something is not working, you can debug individual components.

On the server:

  1. Start xinit in the foreground mode and check if there are no errors.
  2. Start x11vnc in a separate terminal window, and check there are no errors.
  3. Check that x11vnc is listening on port 5900 after startup: $ nc localhost 5900

On the client:

  1. If you can adjust the firewall settings, you can check if you can connect to x11vnc directly bypassing noVNC. If you are on mac, do NOT use the default macOS VNC client! We confirmed that Chicken open source VNC client can establish connection under the same conditions where default macOS client cannot.

Swap space

By default there is no swap space allocated on the machines. You can add this using standard linux commands such as these:

sudo fallocate -l 90G /swapfile
sudo chmod 600 /swapfile 
sudo mkswap /swapfile
sudo swapon /swapfile

To make this persist across boots, add this to the /etc/fstab

/swapfile swap swap defaults 0 0

Check your swap status with these commands

sudo swapon --show
sudo free -h