Deploying DataGateway

Below I will describe how I set up datagateway-dataview and datagateway-api on the scigateway-preprod.esc.rl.ac.uk machine.

Server setup

1. Install node & yarn

(Node Instructions)

(Yarn instructions)

2. Install Apache

yum install httpd

3. Install Python3

yum install epel-release
yum install python36 python36-pip

4. Install mod_wsgi

yum install httpd-devel
pip3 install mod-wsgi
mod_wsgi-express install-module > /etc/httpd/conf.modules.d/02-wsgi.conf

5. Open firewall ports

iptables -A INPUT -p tcp -m tcp --dport 5000 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 5001 -j ACCEPT
iptables-save > /etc/sysconfig/iptables

6. Start Apache

systemctl start httpd

7. Install ICAT stack

Follow the instructions according to the ICAT Manual

Deploying datagateway-dataview

On scigateway-preprod.esc.rl.ac.uk, the code for datagateway and datagateway-api is cloned and built by the ICAT glassfish user.

1. Build datagateway-dataview

git clone https://github.com/ral-facilities/datagateway.git
cd datagateway
yarn install
cd packages/datagateway-dataview
yarn build

This will build datagateway-dataview in production mode. This will minimise the JavaScript and perform other performance improvements, as well as building it in such a way that the parent app can load it.

2. Copy build contents to web folder

cp build/* /var/www/datagateway-dataview/

3. Set up apache virtualhost

/etc/httpd/conf.d/datagateway-dataview.conf

Listen 5001
<VirtualHost *:5001>
    DocumentRoot "/var/www/datagateway-dataview"
    ServerName http://scigateway-preprod.esc.rl.ac.uk

    <Directory /var/www/datagateway-dataview>
        RewriteEngine off

        Options Indexes FollowSymLinks
        AllowOverride None
        Require all granted
    </Directory>
</VirtualHost>

If using HTTPS, the following config should be used instead:

Listen 5001
<VirtualHost *:5001>
    SSLEngine On
    SSLCertificateFile /etc/pki/tls/certs/scigateway-preprod_esc_rl_ac_uk.crt
    SSLCertificateKeyFile /etc/pki/tls/private/scigateway-preprod.key
    SSLCertificateChainFile /etc/pki/tls/certs/scigateway-preprod_esc_rl_ac_uk.ca-bundle.crt
    Header always set Strict-Transport-Security "max-age=63072000"

    DocumentRoot "/var/www/datagateway-dataview"
    ServerName http://scigateway-preprod.esc.rl.ac.uk

    <Directory /var/www/datagateway-dataview>
        RewriteEngine off

        Options Indexes FollowSymLinks
        AllowOverride None
        Require all granted
    </Directory>
</VirtualHost>

systemctl restart httpd

This sets up Apache on port 5001, and tells it to serve the contents of /var/www/datagateway-dataview

4. Copy contents of settings.json to /var/www/html/settings.json

Currently, datagateway-dataview expects the settings file to be in the web root, but SciGateway settings file is stored there. For now, copying in the contents of datagateway-dataview's settings file into SciGateway's is sufficient to get it working. Ensure that api url points to scigateway-preprod.esc.rl.ac.uk:5000

Deploying datagateway-api

In order for datagateway-dataview to be able to load data, it needs to be able to contact the api server.

1. Clone datagateway-api

git clone https://github.com/ral-facilities/datagateway-api.git

2. Add VirtualHost config to Apache:

/etc/httpd/conf.d/datagateway-api.conf

Listen 5000
<VirtualHost *:5000>
     ServerName http://scigateway-preprod.esc.rl.ac.uk

     WSGIPassAuthorization On
     WSGIDaemonProcess datagateway-api user=glassfish group=glassfish threads=1 python-path=/home/glassfish/scigateway/datagateway-api
     WSGIScriptAlias / /var/www/datagateway-api/datagateway-api.wsgi process-group=datagateway-api application-group=%{GLOBAL}

     <Directory /var/www/datagateway-api>
          Options FollowSymLinks
          AllowOverride None
          Require all granted
     </Directory>
</VirtualHost>

This sets up Apache to run mod_wsgi on port 5000. It expects a wsgi file in /var/www/datagateway-api/datagateway-api.wsgi and runs the server as the unprivileged glassfish user.

3. Set up wsgi file

/var/www/datagateway-api/datagateway-api.wsgi

#! /usr/bin/python3.6

import logging
import sys
logging.basicConfig(stream=sys.stderr)
sys.path.insert(0, '/home/glassfish/scigateway/datagateway-api/src/')

from main import app as application

systemctl restart httpd

This tells mod_wsgi the actual location of our app and how to run it.

4. Add data to ICAT

In order to be able to run e2e tests against this server, we need to generate some test data. The command below will run the database generator script and will populate ICAT with test data. In your datagateway-api repo, run the following command:

python -m util.icat_db_generator

5. (OPTIONAL - if you need to be able to test file download functionality) Add some test files

In order to be able to test download functionality, the files need to actually exist so the IDS can serve them.

First, you need to allow the IDS access to ICAT (see https://github.com/icatproject/icat.manual/issues/11). In the IDS's run.properties file you need to change the reader config option to:

reader = simple username root password pw

Additionally, to make things simpler, set the IDS to work in single level mode by commenting out all of the options below the # Properties for archive storage comment.

# Properties for archive storage
!plugin.archive.class = org.icatproject.ids.storage.ArchiveFileStorage
!plugin.archive.dir = /home/glassfish/data/archive
!writeDelaySeconds = 60
!startArchivingLevel1024bytes = 5000000
!stopArchivingLevel1024bytes =  4000000
!storageUnit = datafile
!tidyBlockSize = 500

Reinstall the IDS (rerun ./setup install) and it should now be configured! Now you just need to create files for any files you would like to download - the IDS expects files to be in /home/glassfish/data/main, so create any test files you need there. They can just be empty files that you touch to create.