This project uses Django React/Redux Base Project from Seedstarts as a boilerplate. Please refer to their github for the complete list of technologies used.
Here are the main tools whose knowledge is useful to contribute:
Frontend
- React
- React Router Declarative routing for React
- Webpack for bundling
- Redux Predictable state container for JavaScript apps
- React Router Redux Ruthlessly simple bindings to keep react-router and redux in sync
- styled-components Keeping styles and components in one place
- font-awesome-webpack to customize FontAwesome
Backend
- Django
- Django REST framework Django REST framework is a powerful and flexible toolkit for building Web APIs
- MongoEngine Python ODM for MongoDB
- Celery Distributed tasks queue
This app has a bit complicated architecture due to the nature of other systems it has to communicate.
- MongoDB - a NoSql database where we keep application data in json like documents. Here we also store actions for Unified.
- Oracle - we only read failed workflows names which are stored here by Unified. It would be nice if Unified provided api or at least stored it in MongoDB.
- RequestManager - rest api used for fetching related workflows PrepID name. To access this api you need grid certificate.
- WmStats - rest api from which we get all the info about workflow and errors. To access this api you need grid certificate.
Production/development setup uses nginx as reverse proxy and Gunicorn as an application server. Below are the steps needed to setup environment on RHEL from scratch. Instructions are based on this article How To Set Up Django with Postgres, Nginx, and Gunicorn on CentOS 7
Use these instructions to setup a new production environment from scratch. By following these instructions you will create a dedicated user wtc-console for running this application with Gunicorn, update proxy config for Nginx and setup firewall to allow traffic on port 80.
- Python >=2.7
If you are using CERN managed machine then ask administrator to install latest Node version. If you are managing this machine, then follow this guide for RHEL
-
Go to Oracle client site and download Oracle Instant Client 12.2 Basic. Note: you will need to have Oracle account and download it to your machine.
-
Upload it to the server in your preferred way
-
Run
sudo yum localinstall oracle-instantclient* --nogpgcheck
Add RabbitMQ repo according to RabbitMQ installation guide. Then run these commands:
sudo yum install rabbitmq-serversudo chkconfig rabbitmq-server onsudo /sbin/service rabbitmq-server start
sudo useradd wtc-consolesudo su - wtc-console
Clone this project to wtc-console users home directory.
git clone https://github.com/CMSCompOps/wtc-console.gitcd wtc-console/
Open wtc-console users .bashrc file:
vim ~/.bashrc
Add these lines to it:
export ORACLE_HOME=/usr/lib/oracle/12.2/client64
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export PATH=$PATH:$ORACLE_HOME/bin
And apply these changes:
. ~/.bashrc
pip install --upgrade pippip install virtualenvvirtualenv wtc-console-env
Log out of wtc-console users session and with your user install nginx and dependencies
sudo yum install epel-releasesudo yum install nginx
Add following lines to /etc/nginx/nginx.conf as a first server entry in http section and change domain_name to the actual domain name probably in format of node_name.cern.ch
server {
listen 80;
server_name domain_name;
location = /favicon.ico { access_log off; log_not_found off; }
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://0.0.0.0:8000;
}
}
Test if configuration is valid
sudo nginx -t
Start nginx and set it to run on startup
On RHEL and CentOS:
sudo systemctl start nginxsudo systemctl enable nginx
On Scientific Linux CERN:
sudo service nginx startsudo chkconfig nginx on
If when opening domain.cern.ch you see this error in /var/log/nginx/error.log:
*2 connect() to 127.0.0.1:8000 failed (13: Permission denied) while connecting to upstream, client: some_ip, server: some_domain, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8000/", host: "some_domain"
Then use this command to solve it:
sudo setsebool -P httpd_can_network_connect 1
It turns on httpd connections and -P makes it persistent.
sudo chown -R wtc-console:nginx /home/wtc-console/wtc-consolesudo chmod 770 /home/wtc-console/wtc-console
Ask system administrators to include port 80 to puppet config. If puppet is not used, then you can configure firewall yourself to bypass traffic on this port with these commands:
sudo iptables -I INPUT 1 -p tcp --dport 80 -m state --state NEW,ESTABLISHED -j ACCEPTsudo iptables -I OUTPUT 1 -p tcp --dport 80 -m state --state NEW,ESTABLISHED -j ACCEPTsudo service iptables savesudo service iptables restart
You can see current config with sudo iptables --line -vnL or sudo less /etc/sysconfig/iptables
-
Copy
src/djangoreactredux/settings/local_template.pysetting file tosrc/djangoreactredux/settings/local.pyand modify oracle db credentials throughUNIFIED_DBandMONGODB_DATABASESto reflect your development environment. Errors and debug information concerning celery workers and oracle db configuration will not be rendered through Django, instead always checklogs/celery.logfor errors:grep ERROR logs/celery.log. -
Modify
TNS_ADMINlocated atbin/start_dev.sh, at cern use:/afs/cern.ch/project/oracle/admin/.
Create prod.py in src/djangoreactredux/settings/ directory by using prod_template.py settings template file and update the oracle and mongo db fields with prod values.
cp src/djangoreactredux/settings/prod_template.py src/djangoreactredux/settings/prod.pyvim src/djangoreactredux/settings/prod.py
Create certificates for this machine and copy them to cert/ directory.
Proceed to Development on dev machine or Production deployment steps depending on the machine purpose.
Development on remote node requires to run three sessions (terminals). First one will have frontent watch running, second will have backend running. And third one is for developent with editor of your choise.
Modify src/djangoreactredux/settings/base.py to include the assigned hostname:
ALLOWED_HOSTS = ['localhost','myDevTestMachine.cern.ch']
Become wtc-console user:
sudo su - wtc-console
Go to project dir
cd wtc-console
Start frontend resources watching:
npm install && npm run dev
Start Django backend and Celery workers:
./bin/start_dev.sh
Edit sources with an editor of you choise.
To install CERN Certification Authority and User certificates:
mkdir src/djangoreactredux/cert && cd wtc-console/src/djangoreactredux/cert
wget -c "https://cafiles.cern.ch/cafiles/certificates/CERN%20Root%20Certification%20Authority%202.crt" -O CERNRootCertificationAuthority2.crt
And, depending on the location of your Key/Cert files:
cp ~/.globus/cert.pem crt.pem
cp ~/.globus/key.pem .
The location of these files is specified via src/djangoreactredux/settings/dev.py
Become wtc-console user:
sudo su - wtc-console
Go to project dir
cd wtc-console
Deployment is done with one bash command. It will:
- shutdown celery workers
- shutdown current application if running
- pull latest changes from repository master branch
- install missing dependencies
- build frontend app
- start the application
- start celery workers
./bin/deploy_prod.sh
If for some reason application should be stoppet then use this script:
./src/bin/stop_prod.sh
It will stop Gunicorn and Celery tasks
Logs are in /home/wtc-console/wtc-console/logs
When developing a new feature create your own branch and push your changes at least daily.
Do not push directly to master. Create pull requests and assign someone to approve it. Go through your pull request your self, it helps to see if there is unwanted or commented-out code.
While working on project you might encounter situations where you want to add functionality from third parties. This is done by adding dependecies to external libraries.
Intall dependency with yarn
yarn add dependency-name
Install dependency with pypi
pip install dependency-name