Deploy Xcache via a Singularity Container

Build a Singularity container image with Xcache built-in

xcache.singularity.def can be used to build a Singularity image with Xcache inside and preconfigured. For Singularity 2.4, the command is

sudo singularity build -w xcache.img xcache.singularity.def

(for older versions of Singularity, the build commands maybe different but the recipe file (xcache.singularity.def) will likely be the same). The -w tells Singularity to build an ext3 based image, as opposed to the default, smaller and read-only squashfs based images. An ext3 image can be used with Singularity 2.3 and run on both RHEL6/CentOS6 and 7 platform.

The Singularity recipe file expects the following directories to be available

/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/current/el7-x86_64/etc/grid-security
/cvmfs/oasis.opensciencegrid.org/mis/osg-wn-client/current/el7-x86_64/etc/vomsdir

They will be used to populate the /etc/grid-security/certificates,vomsdir} directories. If they are not available, then those directories will be left empty. (This can be change in the recipe file - for example to use build host's /etc/grid-security directory).

Run the container

At minimum, one need to provide a dedicated file system that is fully writable by the user who run the container. The filesystem will be bind mounted to /data. The command is

 singularity run -B /my_filesystem:/data xcache.img

With Singularity 2.3 and older, -H ${HOME}:/mnt may be needed. If you only have network filesystem available for cache space, using a loop mounted file system image may give you better performance.

In some cases (and most ATLAS use cases), user will need to provide and continuously update a X509 user proxy certificate. The container will search the default location /tmp/x509up_u$(id -u) for x509 proxy.

The above command will immediately return, leaving a xrootd process behind. Using ps -ef one can see the xrootd process (with ppid=1) but not the Singularity process. This is expected. To stop the xcache, type pkill xrootd.

Bind mount options

The container can utilize the following bind mounts:

• /var/run/x509up: In case your x509 user proxy certificate isn't in the standard location (above), you can bind mount it to /var/run/x509up.
• /etc/grid-security/{certificates, vomsdir}: can be overwritten by bind mounts.
• /etc/xrootd/xcache.cfg: the container will automatically generated (and save) a configuration file to /tmp/xcache.cfg. The container will give a best guess about how much memory should be used by the cache, and the low and high water marks for disk space cleaning. You can modify it and then bind mount your own configure file to /etc/xrootd/xcache.cfg.

Start a cluster of Xcache

Just like regular xrootd, Xcache can be a cluster. To start the cluster, you need to define a unix environment variable XCACHE_RDR to the full DNS name of the redirector host (hostname -f). Do this on all machines: redirector and data servers. Then run the Singularity command (above) on all of them. This will start both xrootd and cmsd processes on all machines. To stop, type pkill xrootd and pkill cmsd.

Note: on redirector host, bind mount to /data is still needed. Only log files will appear there so the space usage will be small. There is no need to provide x509 user proxy certificate on the redirector host.