Skip to content

Deployment

Jump to bottom
Nathan Kim edited this page Aug 20, 2023 · 4 revisions

AWS Deployment Setup Guide

Overview

We trigger deployments from Github Actions, configured in the script .github/workflows/cd.yaml. Once triggered, a state of the repository is saved as a deploy on AWS S3, and an AWS CodeDeploy run is triggered to deploy our application on an AWS EC2 instance.

Our CodeDeploy configuration can be partially viewed at appspec.yml, but configuration mostly exists on the AWS console. There, we have configured a "blue-green" deployment setup; when a deployment is triggered from AWS CodeDeploy, an existing "blue" instance remains running until a "green" instance has deployed. Then, an intermediary server called a load balancer switches any user connections to evictorbook.com to the new server, and shuts the existing "blue" server down. The load balancer also holds a certificate from AWS Certificate Manager, allowing the certificate to persist even while the "blue" or "green" servers are ephemeral. Through this configuration, we can deploy new versions of Evictorbook without downtime for the user, are safe from downtime or stressful bugfixing sessions in case the deployment fails, and can avoid limits to certificate renewal (explained more below).

Steps to replicate the blue-green deployment

  1. Create an AMI that includes Docker and the CodeDeploy agent, a launch template from that AMI, and an auto-scaling group from that launch template. These will be used by CodeDeploy to automatically deploy instances.
    • The launch template should allow traffic on port 80 and give permissions to the IAM user that CodeDeploy runs on.
    • The auto-scaling group should have at minimum 1 server and as maximum can have any amount, but we recommend 2 servers to conserve costs.
  2. Create a load balancer and a target group. The target group should be set to HTTP and the load balancer should have an HTTPS listener that forwards traffic to the target group. You should also have an HTTP listener forwarding traffic to the HTTPS listener.
    • We recommend turning off the health checks at first -- the health checks will fail at first because port 80 will not serve anything until after the first deploy, which can result in continual deregistrations and creations of new instances in the auto-scaling group, and can potentially result in failed deployments.
    • Also in the target group, we recommend setting a shorter deregistration delay than the default. We use 15 seconds. This can help speed up deployments.
  3. Create an application and deployment group on CodeDeploy. If using our Github Actions and start.sh scripts, the deployment group name should be "ebook-lb". The deployment group should be set to a blue-green deployment, the target group set to the HTTP target group you created above, and the auto-scaling group you created above should be specified as well.
    • We recommend setting the "blue" instance termination delay to 5 minutes or shorter instead of the default of 1 hour; otherwise both the new successful instance and the old, non-functional instance will remain running at the same time.

Launching an instance to deploy the application

We have an Amazon Machine Image (AMI) and an associated launch template that can automate many of the steps below.

  1. Launch an EC2 instance running Ubuntu (e.g. Ubuntu 18.04). We recommend using a t2.large size instance, as it is a relatively cost-friendly instance size that can still support our database.

  2. Make sure the security group for the EC2 instance has port 3000 open

  3. SSH into the instance

  4. Install Docker (Linux)

    sudo apt-get -y install docker-engine jq awscli
sudo usermod -aG docker $USER
sudo reboot

  5. Export the test/production Neo4j endpoint as the environment variable DB_ENDPOINT.
    Be sure it references a current neo4j instance. E.g.

    export DB_ENDPOINT=ec2-01-234-56-789.us-west-2.compute.amazonaws.com

    To remove the the environment variable (for example when switching back to a dev environment), unset it

    unset DB_ENDPOINT

  6. Run the production Docker-Compose. This will launch 3 containers: the UI, the api server and the nginx webserver.

    docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d

Deployment Scripts Overview

AWS runs these scripts when a deployment is triggered. The main job of these scripts is to run docker-compose to create the staging or production environment from scratch.

Nginx Configuration Guide

Nginx is web server software that receives and responds to HTTP/HTTPS requests.

We have two instances of nginx running at the same time. There is the proxy, which is configured per the configuration files located in this folder. The proxy is the "edge" instance that forwards traffic to and from the UI and API containers to web browsers over the Internet. The UI container runs a different instance of nginx to serve static content in the production environment. In a local dev environment, a temporary web server is set up by by create-react-app which serves the content dynamically for development purposes.

The API container, rather than using nginx, uses the node process to serve content dynamically using the web server package called Express. These responses are first sent through the proxy before reaching the client.

The nginx proxy does the following:

  • encrypts responses using SSL cert files
  • serves HTTP content to prove the server's identity to LetsEncrypt
  • forwards data compression added either by UI's nginx or API's Express
  • adds Cache-

SSL Cert Overview

The SSL cert files need to exist at the location where nginx looks for them when it starts up, and as long as the cert is valid the site will use SSL. Letsencrypt is a free certificate authority that issues SSL certs, and those certs enable HTTPS and prevent browsers from showing a warning screen. Letsencrypt certs last 90 days. There is a limit to 5 reissuances of a cert per week, and there is no way to extend this limitation.

Evictorbook until recently had logic, to be run with each deploy of our application, to request a renewal of our SSL certificate from LetsEncrypt. We would create a temporary "dummy" certificate to prevent Nginx from crashing, place files from the certbot Docker image on our Nginx proxy so that LetsEncrypt could for the existence of these files and thus verify that we own the domain, and LetsEncrypt would then validate our ownership of the domain and issue a certificate. The docker-compose script loaded a small certbot docker image that attempted to renew the cert every 12 hours. Renewing a cert in this manner does not cause it to be reissued, so this did not count toward the limit.

We moved away from this as we had adopted a load balancer from AWS to facilitate blue-green deployments as explained above. As part of this deployment pattern, we have a new service called a load balancer that always runs even when individual servers are taken down. We attached a certificate to this load balancer, eliminating the need for our own logic to renew the certificate or attach it to a particular EC2 instance.

To do so, we took the following steps:

  1. Request a certificate from AWS Certificate Manager for evictorbook.com
  2. On Route53, the AWS service for configuring domains, we created a CNAME record that the certificate manager uses to validate the certificate and ensure we own the domain
  3. Also on Route53, we created an alias record to direct users visiting evictorbook.com to the load balancer
  4. We attached the certificate from AWS Certificate Manager to the load balancer, so that our application can serve HTTPS requests to evictorbook.com.
  5. We created redirects so that users visiting http://evictorbook.com, sf.evictorbook.com or oakland.evictorbook.com, are redirected towards https://evictorbook.com.

Note: this documentation is a work in progress

Clone this wiki locally