fda-mystudies/deployment at v2.0.6 · GoogleCloudPlatform/fda-mystudies

Name	Name	Last commit message	Last commit date
parent directory ..
kubernetes	kubernetes
scripts	scripts
terraform	terraform
README.md	README.md
deployment.hcl	deployment.hcl
mystudies.hcl	mystudies.hcl

Deploying FDA MyStudies

Introduction

This guide provides instructions for semi-automated deployment of FDA MyStudies to Google Cloud. It is designed to be completed in just a few hours.

This document explains how to integrate Terraform’s open-source infrastructure-as-code with continuous integration and continuous deployment (CICD) for a reproducible and easy-to-maintain environment. Kubernetes has been selected as the open-source container orchestration tool for its robust scaling and cluster management. All necessary Terraform templates, setup scripts and Kubernetes configuration files are included in the repository. This guide will explain what cloud services to enable, which configuration parameters to update and provide step-by-step instructions for each manual portion of the process.

This approach to deployment is based on Google Cloud's HIPAA-aligned architecture, which is designed to simplify your compliance journey. You can learn more about this approach to compliance in the Setting up a HIPAA-aligned project solution guide.

Deployment overview

Following this guide will result in your own unique instance of the FDA MyStudies platform. The resulting deployment will have the structure illustrated in Figure 1. Each functionally distinct aspect of the platform is deployed into its own cloud project to facilitate compartmentalization and robust access management. Each project and resource is named for its purpose, and has a {PREFIX}-{ENV} label, where {PREFIX} is a consistent name of your choice and {ENV} describes your environment (for example, dev or prod). The list of projects you will create for your deployment is as follows:

Project	Name	Purpose
Devops	`{PREFIX}-{ENV}-devops`	This project executes the Terraform CICD pipeline that keeps your infrastructure aligned with the state defined in the `deployment/terraform/` directory of your GitHub repository
Apps	`{PREFIX}-{ENV}-apps`	This project stores the container images for each of your FDA MyStudies applications, updates those images with CICD pipelines that monitor changes you make to the application directories of your GitHub repository, and administers the Kubernetes cluster that operates those images (Figure 2 diagrams each the applications and how they related to their data sources)
Data	`{PREFIX}-{ENV}-data`	This project contains the MySQL databases that support each of the FDA MyStudies applications, and the blob storage buckets that hold study resources and consent documents
Firebase	`{PREFIX}-{ENV}-firebase`	This project contains the NoSQL database that stores the study response data
Networks	`{PREFIX}-{ENV}-networks`	This project manages the network policies and firewalls
Secrets	`{PREFIX}-{ENV}-secrets`	This project manages the deployment’s secrets, such as client ids and client secrets
Audit	`{PREFIX}-{ENV}-audit`	This project stores the audit logs for the FDA MyStudies platform and applications

This deployment configures the applications URLs as follows:

Application	URL	Notes
Study builder	`studies.{PREFIX}-{ENV}.{DOMAIN}/study-builder`	This URL navigates an administrative user to the `Study builder` user interface
Study datastore	`studies.{PREFIX}-{ENV}.{DOMAIN}/study-datastore`	This URL is for the `Study datastore` back-end service
Participant manager	`participants.{PREFIX}-{ENV}.{DOMAIN}/participant-manager`	This URL navigates an administrative user to the `Participant manager` user interface
Participant manager datastore	`participants.{PREFIX}-{ENV}.{DOMAIN}/participant-manager-datastore`	This URL is for the `Participant manager datastore` back-end service
Participant datastore	`participants.{PREFIX}-{ENV}.{DOMAIN}/participant-user-datastore` `participants.{PREFIX}-{ENV}.{DOMAIN}/participant-enroll-datastore` `participants.{PREFIX}-{ENV}.{DOMAIN}/participant-consent-datastore`	These URLs are for the `Participant datastore` back-end services
Response datastore	`participants.{PREFIX}-{ENV}.{DOMAIN}/response-datastore`	This URL is for the `Response datastore` back-end service
Auth server	`participants.{PREFIX}-{ENV}.{DOMAIN}/auth-server`	This URL is for the administrative users and study participants to log into their respective applications
Hydra	`participants.{PREFIX}-{ENV}.{DOMAIN}/oauth2`	This URL is used by the `Auth server` to complete OAuth 2.0 consent flows

More information about the purpose of each application can be found in the Platform Overview guide. Detailed information about configuration and operation of each application can be found in their respective READMEs.

Figure 1: Overall architecture of the semi-automated deployment

Figure 2: Application architecture

The deployment process takes the following approach:

Create a copy of the FDA MyStudies repository that you will use for your deployment
Create the devops cloud project that will be used to orchestrate your deployment
Connect your cloned FDA MyStudies repository to your devops project and set up the CICD pipelines that will automate the rest of your deployment
Provision the necessary cloud resources using your CICD pipelines
Set up a second CICD pipeline that will automate creation of your application containers
Create a Kubernetes cluster to run your application containers
Create your initial user accounts and configure the required certificates, secrets, URLs, policies and network mesh
Customize branding and text content as desired
Create your first study
Configure and deploy your mobile applications

Before you begin

Familiarize yourself with:
- Terraform and Terraform Engine
- Kubernetes and Google Kubernetes Engine
- CICD and Google Cloud Build
- IAM and Google Cloud’s resource hierarchy
Understand how the Terraform config files and cloud resources are named and organized in the deployment:
- {PREFIX} is a name you choose for your deployment that will be prepended to various directories, cloud resources and URLs (for example this could be ‘mystudies’)
- {ENV} is a label you choose that will be appended to {PREFIX} in your directories and cloud resources (for example this could be ‘dev’, ‘test’ or ‘prod’)
- {DOMAIN} is the domain you will be using for your URLs (for example, ‘your_company_name.com’ or ‘your_medical_center.edu’)
- /deployment/deployment.hcl is the file where you will specify top-level parameters for your deployment (for example, the values of {PREFIX}, {ENV} and {DOMAIN})
- /deployment/mystudies.hcl is the file that represents the overall recipe for the deployment (you will uncomment various aspects of this recipe as your deployment progresses)
- The directories created in /deployment/terraform/ by the tfengine command represent distinct cloud projects that the CICD pipeline monitors to create, update or destroy resources based on the changes you make to those directories
- The other directories in the FDA MyStudies repository map to the various components of the platform and contain Terraform and Kubernetes configuration files, such as tf-deployment.yaml and tf-service.yaml, that support each component’s deployment

Prepare the cloud platform

Make sure you have access to a Google Cloud environment that contains an organization resource (if you don’t have an organization resource, you can obtain one by creating a Google Workspace and selecting a domain)
Confirm the billing account that you will use has quota for 10 or more projects (newly created billing accounts may default to a 3-5 project quota)
- You can test how many projects your billing account can support by manually creating projects and linking them to your billing account, if you are able to link 10 projects to your billing account then you can proceed, otherwise request additional quota (don’t forget to unlink the test projects from your billing account, otherwise your quota may be exhausted)
Use the resource manager to create a folder to deploy your FDA MyStudies infrastructure into, for example you could name this folder {PREFIX}-{ENV} (if you do not have the resourcemanager.folderAdmin role for your organization, you may need to ask your Google Cloud IT administrator to do this for you)
Confirm you have access to a user account with the following Cloud IAM roles:
- roles/resourcemanager.folderAdmin for the folder you created
- roles/resourcemanager.projectCreator for the folder you created
- roles/compute.xpnAdmin for the organization (note, this permission must be at the organization level not the folder level)
- roles/billing.admin for the billing account that you will use

Use the groups manager to create the following administrative IAM groups that will be used during deployment:

Group name	Description
`{PREFIX}-{ENV}-folder-admins@{DOMAIN}`	Members of this group have the resourcemanager.folderAdmin role for your deployment’s folder (for example, a deployment with prefix `mystudies`, environment `prod` and domain `example.com`, would require a group named `[email protected]`)
`{PREFIX}-{ENV}-devops-owners@{DOMAIN}`	Members of this group have owners access for the devops project, which is required to make changes to the CICD pipeline and Terraform state
`{PREFIX}-{ENV}-auditors@{DOMAIN}`	Members of this group have the `iam.securityReviewer` role for your deployment’s folder, and the `bigquery.user` and `storage.objectViewer` roles for your audit log project
`{PREFIX}-{ENV}-cicd-viewers@{DOMAIN}`	Members of this group can view CICD results in Cloud Build, for example the results of the `terraform plan` presubmit and `terraform apply` postsubmit
`{PREFIX}-{ENV}-bastion-accessors@{DOMAIN}`	Members of this group have permission to access the bastion host project, which provides access to the private Cloud SQL instance
`{PREFIX}-{ENV}-project-owners@{DOMAIN}`	Members of this group have owners access to each of the deployment’s projects

Add the user account that you will be using for deployment to these groups (if it is not already a member)

Set up your environment

You can work in an existing environment, or you can configure a new environment by creating a VM instance in the Google Cloud project of your choice (for example, an e2-medium GCE VM with Debian GNU/Linux 10 and default settings)

Confirm you have the following dependencies installed and added to your $PATH:

Install the Google Cloud command line tool gcloud (already installed if using a Google Compute Engine VM), for example:
```
apt-get install google-cloud-sdk
```
Install the Cloud Storage command line tool gsutil (already installed if using a Google Compute Engine VM)

Install the Kubernetes command line tool kubectl, for example:

sudo apt-get update && sudo apt-get install -y apt-transport-https gnupg2 curl && \
  curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && \
  echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list && \
  sudo apt-get update && \
  sudo apt-get install -y kubectl

Install Terraform 0.12.29, for example:

sudo apt-get install software-properties-common -y && \
  curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add - && \
  sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main" && \
  sudo apt-get update && sudo apt-get install terraform=0.12.29

Install Terraform Engine, for example:

VERSION=v0.4.0 && \
  sudo apt install wget -y && \
  sudo wget -O /usr/local/bin/tfengine https://github.com/GoogleCloudPlatform/healthcare-data-protection-suite/releases/download/${VERSION}/tfengine_${VERSION}_linux-amd64 && \
  sudo chmod +x /usr/local/bin/tfengine

Install Git, for example:
```
sudo apt-get install git
```

Duplicate the FDA MyStudies repository, then clone it locally
Update /deployment/deployment.hcl with the values for your deployment
Update /deployment/scripts/set_env_var.sh for your deployment, then use the script to set your environment variables, for example:
```
source set_env_var.sh    # executed from your /deployment/scripts directory
```
Authenticate as a user with the permissions described above (this deployment assumes gcloud and Terraform commands are made as a user, rather than a service account)
- Login and update your application default credentials, for example you could run gcloud auth login --update-adc (when using a Google Compute Engine VM you must update the application default credentials, otherwise requests will continue to be made with its default service account)
- Remember to log your user account out once your deployment is complete

Create your devops project and configure CICD pipelines

Generate your Terraform configuration files
- Set the enable_gcs_backend flag in mystudies.hcl to false, for example:
```
sed -e 's/enable_gcs_backend = true/enable_gcs_backend = false/g' \
 -i.backup $GIT_ROOT/deployment/mystudies.hcl
```
- Execute the tfengine command to generate the configs (by default, CICD will look for Terraform configs under the deployment/terraform/ directory in the GitHub repo, so set the --output_path to point to the deployment/terraform/ directory inside the local root of your GitHub repository), for example:
```
tfengine --config_path=$ENGINE_CONFIG --output_path=$GIT_ROOT/deployment/terraform
```
Use Terraform to create the {PREFIX}-{ENV}-devops project and Terraform state bucket (if this step fails, confirm you have updated your application default credentials and that the required version of Terraform is installed), for example:
```
cd $GIT_ROOT/deployment/terraform/devops
terraform init && terraform apply
```

Backup the state of your {PREFIX}-{ENV}-devops project to the newly created state bucket by setting the enable_gcs_backend flag in mystudies.hcl to true and regenerating the Terraform configs, for example:

sed -e 's/enable_gcs_backend = false/enable_gcs_backend = true/g' \
  -i.backup $GIT_ROOT/deployment/mystudies.hcl    
tfengine --config_path=$ENGINE_CONFIG --output_path=$GIT_ROOT/deployment/terraform
cd $GIT_ROOT/deployment/terraform/devops
terraform init -force-copy

Open Cloud Build in your new {PREFIX}-{ENV}-devops project and connect your cloned GitHub repository (skip adding triggers as Terraform will create them in the next step)
Create the CICD pipeline for your deployment (this will create the Cloud Builder triggers that will run whenever a pull request containing changes to files in $GIT_ROOT/deployment/terraform/ is raised against the GitHub branch that you specified in deployment.hcl), for example:
```
cd $GIT_ROOT/deployment/terraform/cicd
terraform init && terraform apply
```
The Cloud Build service account will need to be a Shared VPC Admin (XPN Admin) at the organization level as this permission cannot be granted at the folder level.
1. Open Cloud IAM in your {PREFIX}-{ENV}-devops project, and copy the Member name of the service account with the role Cloud Build Service Agent. The format should be ############@cloudbuild.gserviceaccount.com
2. At the top of the page change from the {PREFIX}-{ENV}-devops project to the organization containing your folder and projects.
3. Click Add to add a new member to the organization
4. Enter the Cloud Build service account in the New members field and add the role Compute Shared VPC Admin and click Save.

Deploy your platform infrastructure

Commit your local git working directory (which now represents your desired infrastructure state) to a new branch in your cloned FDA MyStudies repository, for example using:
```
cd $GIT_ROOT
git checkout -b initial-deployment
git add $GIT_ROOT/deployment/terraform
git commit -m "Perform initial deployment"
git push origin initial-deployment
```
Trigger Cloud Build to run the Terraform pre-submit checks by using this new branch to create a pull request against the branch you specified in deployment.hcl (you can view the status of your pre-submit checks and re-run jobs as necessary in the Cloud Build history of your devops project)
Once your pre-submit checks have completed successfully, and you have received code review approval, merge your pull request into the main branch to trigger the terraform apply post-submit operation (this operation may take up to 45 minutes - you can view the status of the operation in the Cloud Build history of your devops project)

Note: If your pre-submit checks or post-submit terraform apply fail with an error related to billing accounts, you may not have the quota necessary to attach all of your projects to the specified billing account. You may need to request additional quota.
Grant the roles/owner permission to the {PREFIX}-{ENV}-project-owners@{DOMAIN} group for each of your newly created projects

Configure your domain for the deployment

Determine the name servers that Cloud DNS has allocated to your {PREFIX}-{ENV} subdomain
- Navigate to DNS zones in your {PREFIX}-{ENV}-apps project
- Click on the zone named {PREFIX}-{ENV}, then click Registrar Setup in the upper right to view the name servers allocated to your subdomain
Update your domain’s DNS settings with your domain registrar to create a delegated subzone with the name servers allocated by Cloud DNS (this process differs across domain registrars - consult your domain registrar’s documentation to determine how to make these changes, or ask your domain’s IT administrator for help)
- If your domain registrar is Google Domains, you can create the delegated subzone as follows:
  1. Log into Google Domains using the account that administers your domain
  2. Navigate to the DNS page for your domain and scroll down to the ‘custom resource records’ section (you do not need to make changes in the other sections)
  3. Create an NS resource record for your {PREFIX}-{ENV} subdomain that matches the name servers specified in Cloud DNS, for example:
Verify your setup (it may take up to 48 hours for DNS changes to propagate across the internet)

Configure your deployment’s databases

Create a Native mode Cloud Firestore database in your {PREFIX}-{ENV}-firebase project (the location selected here does not need to match the region configured in your deployment.hcl file)
Use Terraform and CICD to create Firestore indexes, a Cloud SQL instance, user accounts and IAM role bindings
- Uncomment the blocks for steps 5.1 through 5.6 in mystudies.hcl, for example:
```
sed -e 's/#5# //g' -i.backup $GIT_ROOT/deployment/mystudies.hcl
```
- Regenerate the Terraform configs and commit the changes to your repo, for example:
```
cd $GIT_ROOT
tfengine --config_path=$ENGINE_CONFIG --output_path=$GIT_ROOT/deployment/terraform
git checkout -b database-configuration
git add $GIT_ROOT/deployment/terraform
git commit -m "Configure databases"
git push origin database-configuration
```
- Once your pull request pre-submit checks have completed successfully, and you have received code review approval, merge your pull request to trigger terraform apply(this may take up to 20 minutes - you can view the status of the operation in the Cloud Build history of your devops project)
Configure the permissions of your SQL script bucket so that your Cloud SQL instance can import the necessary initialization scripts
- Uncomment the blocks for Steps 6 in mystudies.hcl, for example:
```
sed -e 's/#6# //g' -i.backup $GIT_ROOT/deployment/mystudies.hcl
```
- Regenerate the Terraform configs and commit the changes to your repo, for example:
```
cd $GIT_ROOT
tfengine --config_path=$ENGINE_CONFIG --output_path=$GIT_ROOT/deployment/terraform
git checkout -b sql-bucket-permissions
git add $GIT_ROOT/deployment/terraform
git commit -m "Set SQL bucket permissions"
git push origin sql-bucket-permissions
```
- Once your pull request pre-submit checks have completed successfully, and you have received code review approval, merge your pull request to trigger terraform apply(this may take up to 10 minutes - you can view the status of the operation in the Cloud Build history of your devops project)

Initialize your MySQL databases by importing SQL scripts

Upload the necessary SQL script files to the {PREFIX}-{ENV}-mystudies-sql-import storage bucket that you created during Terraform deployment, for example:

gsutil cp \
  ${GIT_ROOT}/study-builder/sqlscript/* \
  ${GIT_ROOT}/response-datastore/sqlscript/mystudies_response_server_db_script.sql \
  ${GIT_ROOT}/participant-datastore/sqlscript/mystudies_app_info_update_db_script.sql \
  ${GIT_ROOT}/participant-datastore/sqlscript/mystudies_participant_datastore_db_script.sql \
  ${GIT_ROOT}/auth-server/sqlscript/mystudies_oauth_server_hydra_db_script.sql \
  ${GIT_ROOT}/hydra/sqlscript/create_hydra_db_script.sql \
  gs://${PREFIX}-${ENV}-mystudies-sql-import

Import the SQL scripts from cloud storage to your Cloud SQL instance, for example:

gcloud sql import sql --project=${PREFIX}-${ENV}-data mystudies gs://${PREFIX}-${ENV}-mystudies-sql-import/create_hydra_db_script.sql -q
gcloud sql import sql --project=${PREFIX}-${ENV}-data mystudies gs://${PREFIX}-${ENV}-mystudies-sql-import/mystudies_oauth_server_hydra_db_script.sql -q
gcloud sql import sql --project=${PREFIX}-${ENV}-data mystudies gs://${PREFIX}-${ENV}-mystudies-sql-import/HPHC_My_Studies_DB_Create_Script.sql -q
gcloud sql import sql --project=${PREFIX}-${ENV}-data mystudies gs://${PREFIX}-${ENV}-mystudies-sql-import/procedures.sql -q
gcloud sql import sql --project=${PREFIX}-${ENV}-data mystudies gs://${PREFIX}-${ENV}-mystudies-sql-import/version_info_script.sql -q
gcloud sql import sql --project=${PREFIX}-${ENV}-data mystudies gs://${PREFIX}-${ENV}-mystudies-sql-import/mystudies_response_server_db_script.sql -q
gcloud sql import sql --project=${PREFIX}-${ENV}-data mystudies gs://${PREFIX}-${ENV}-mystudies-sql-import/mystudies_participant_datastore_db_script.sql -q

Enable the Cloud SQL Admin API for your {PREFIX}-{ENV}-apps project, for example:

gcloud config set project $PREFIX-$ENV-apps && \
  gcloud services enable sqladmin.googleapis.com

Configure and deploy your applications

Make a request to increase the Global Compute Engine API Backend Services quota for your {PREFIX}-{ENV}-apps project to 20 (if it is not already set at, or beyond, this value)
Enable CICD for the application directories of your cloned GitHub repository so that changes you make to the application code will automatically build the application containers for your deployment
- Enable Cloud Build in your {PREFIX}-{ENV}-apps project and connect your cloned GitHub repository (skip adding triggers as Terraform will create them in the next step)
- Uncomment the Cloud Build triggers portion of the apps project (step 7) in mystudies.hcl, for example:
```
sed -e 's/#7# //g' -i.backup $GIT_ROOT/deployment/mystudies.hcl
```
- Regenerate the Terraform configs and commit the changes to your repo, for example:
```
cd $GIT_ROOT
tfengine --config_path=$ENGINE_CONFIG --output_path=$GIT_ROOT/deployment/terraform
git checkout -b enable-apps-CICD
git add $GIT_ROOT/deployment/terraform
git commit -m "Enable CICD for applications"
git push origin enable-apps-CICD
```
- Once your pull request pre-submit checks have completed successfully, and you have received code review approval, merge your pull request to trigger terraform apply(this may take up to 10 minutes - you can view the status of the operation in the Cloud Build history of your devops project)
Update the Kubernetes and application configuration files with the values specific to your deployment
- Replace the <PREFIX>, <ENV> and <LOCATION> values for each tf-deployment.yaml in your repo, for example:
```
find $GIT_ROOT -name 'tf-deployment.yaml' \
  -exec sed -e 's/<PREFIX>-<ENV>/'$PREFIX'-'$ENV'/g' \
  -e 's/<LOCATION>/'$LOCATION'/g' -i.backup '{}' \;
```
- Replace the <PREFIX>, <ENV> and <DOMAIN> values in /deployment/kubernetes/cert.yaml and /deployment/kubernetes/ingress.yaml, for example:
```
sed -e 's/<PREFIX>/'$PREFIX'/g' \
  -e 's/<ENV>/'$ENV'/g' \
  -e 's/<DOMAIN>/'$DOMAIN'/g' -i.backup \
  $GIT_ROOT/deployment/kubernetes/cert.yaml
sed -e 's/<PREFIX>/'$PREFIX'/g' \
  -e 's/<ENV>/'$ENV'/g' \
  -e 's/<DOMAIN>/'$DOMAIN'/g' -i.backup \
  $GIT_ROOT/deployment/kubernetes/ingress.yaml
```
- In /participant-manager/src/environments/environment.prod.ts, replace <BASE_URL> with your participants.{PREFIX}-{ENV}.{DOMAIN} value and <auth-server-client-id> with the value of your auto-auth-server-client-id secret (you can find this value in the Secret Manager of your {PREFIX}-{ENV}-secrets project), for example:
```
gcloud config set project $PREFIX-$ENV-secrets
export auth_server_client_id=$( \
  gcloud secrets versions access latest --secret="auto-auth-server-client-id")
sed -e 's/<BASE_URL>/participants.'$PREFIX'-'$ENV'.'$DOMAIN'/g' \
  -e 's/<AUTH_SERVER_CLIENT_ID>/'$auth_server_client_id'/g' -i.backup \
  $GIT_ROOT/participant-manager/src/environments/environment.prod.ts
```
- Commit the changes to your repo, for example:
```
cd $GIT_ROOT
git checkout -b configure-application-properties
git add $GIT_ROOT
git commit -m "Initial configuration of application properties"
git push origin configure-application-properties
```
- Once your pull request pre-submit checks have completed successfully, and you have received code review approval, merge your pull request to build your container images, after which they will be available in the Container Registry of your apps project at http://gcr.io/{PREFIX}-{ENV}-apps (this may take up to 10 minutes - you can view the status of the operation in the Cloud Build history of your {PREFIX}-{ENV}-apps project)

Open Secret Manager for your {PREFIX}-{ENV}-secrets project and fill in the values for secrets with the prefix “manual-” (or set your gcloud project with gcloud config set project $PREFIX-$ENV-secrets and use the commands described below - afterwards clear your shell history with history -c)

Manually set secret	Description	When to set	Example command
`manual-mystudies-email-address`	The login of the email account you want MyStudies to use to send system-generated emails	Set this value now or enter a placeholder	`echo -n "<SECRET_VALUE>" \| gcloud secrets versions add "manual-mystudies-email-address" --data-file=-`
`manual-mystudies-email-password`	The password for that email account	Set this value now or enter a placeholder	`echo -n "<SECRET_VALUE>" \| gcloud secrets versions add "manual-mystudies-email-password" --data-file=-`
`manual-mystudies-contact-email-address`	The email address that the in-app contact and feedback forms will send messages to	Set this value now or enter a placeholder	`echo -n "<SECRET_VALUE>" \| gcloud secrets versions add "manual-mystudies-contact-email-address" --data-file=-`
`manual-mystudies-from-email-address`	The return email address that is shown is system-generated messages (for example, `[email protected]`)	Set this value now or enter a placeholder	`echo -n "<SECRET_VALUE>" \| gcloud secrets versions add "manual-mystudies-from-email-address" --data-file=-`
`manual-mystudies-from-email-domain`	The domain of the above email address (just the value after “@”)	Set this value now or enter a placeholder	`echo -n "<SECRET_VALUE>" \| gcloud secrets versions add "manual-mystudies-from-email-domain" --data-file=-`
`manual-mystudies-smtp-hostname`	The hostname for your email account’s SMTP server (for example, `smtp.gmail.com`)	Set this value now or enter a placeholder	`echo -n "<SECRET_VALUE>" \| gcloud secrets versions add "manual-mystudies-smtp-hostname" --data-file=-`
`manual-mystudies-smtp-use-ip-allowlist`	Typically ‘false’; if ‘true’, the platform will not authenticate to the email server and will rely on the allowlist configured in the SMTP service	Set this value to `true` or `false` now (you can update it later)	`echo -n "false" \| gcloud secrets versions add "manual-mystudies-smtp-use-ip-allowlist" --data-file=-`
`manual-log-path`	The path to a directory within each application’s container where your logs will be written (for example `/logs`)	Set this value now	`echo -n "/logs" \| gcloud secrets versions add "manual-log-path" --data-file=-`
`manual-org-name`	The name of your organization that is displayed to users, for example ‘Sincerely, the `manual-org-name` support team’	Set this value now	`echo -n "<SECRET_VALUE>" \| gcloud secrets versions add "manual-org-name" --data-file=-`
`manual-terms-url`	URL for a terms and conditions page that the applications will link to (for example, `https://example.com/terms`)	Set this value now or enter a placeholder	`echo -n "<SECRET_VALUE>" \| gcloud secrets versions add "manual-terms-url" --data-file=-`
`manual-privacy-url`	URL for a privacy policy page that the applications will link to (for example, `https://example.com/privacy`)	Set this value now or enter a placeholder	`echo -n "<SECRET_VALUE>" \| gcloud secrets versions add "manual-privacy-url" --data-file=-`
`manual-mobile-app-appid`	The value of the `App ID` (15 characters max) that you will configure on the Settings page of the Study builder user interface when you create your first study (you will also use this same value when configuring your mobile applications for deployment)	Set now if you know what value you will use when you create your first study - otherwise enter a placeholder and update once you have created a study in the Study builder	`echo -n "<SECRET_VALUE>" \| gcloud secrets versions add "manual-mobile-app-appid" --data-file=-`
`manual-android-bundle-id`	The value of `applicationId` that you will configure in `Android/app/build.gradle` during Android configuration, for example `{PREFIX}_{ENV}.{DOMAIN}` (note that some characters are not permitted)	If you know what value you will use during Android deployment you can set this now, otherwise enter a placeholder and update later (leave as placeholder if you will be deploying to iOS only)	`echo -n "<SECRET_VALUE>" \| gcloud secrets versions add "manual-android-bundle-id" --data-file=-`
`manual-fcm-api-url`	URL of your Firebase Cloud Messaging API (documentation)	Set now if you know what this value will be - otherwise create a placeholder and update after completing your Android deployment (leave as placeholder if you will be deploying to iOS only)	`echo -n "<SECRET_VALUE>" \| gcloud secrets versions add "manual-fcm-api-url" --data-file=-`
`manual-android-server-key`	The Firebase Cloud Messaging server key that you will obtain during Android configuration	Set now if you know what this value will be - otherwise create a placeholder and update after completing your Android deployment (leave as placeholder if you will be deploying to iOS only)	`echo -n "<SECRET_VALUE>" \| gcloud secrets versions add "manual-android-server-key" --data-file=-`
`manual-android-deeplink-url`	The sign-in screen is run on the Hydra-based auth server. This URL is a deep link that helps redirect users to the native mobile app after they sign in. (for example, `app://{PREFIX}-{ENV}.{DOMAIN}/mystudies`)	Set now if you know what this value will be - otherwise create a placeholder and update after completing your Android deployment (leave as placeholder if you will be deploying to iOS only)	`echo -n "<SECRET_VALUE>" \| gcloud secrets versions add "manual-android-deeplink-url" --data-file=-`
`manual-ios-bundle-id`	The value you will obtain from Xcode (Project target > General tab > Identity section > Bundle identifier) during iOS configuration - for a production application, the bundle ID needs to be verified with Apple and is usually a reverse domain name that you own; it is a unique app identifier and application capabilities are mapped to this value (details)	Set now if you know what this value will be - otherwise create a placeholder and update after completing your iOS deployment (leave as placeholder if you will be deploying to Android only)	`echo -n "<SECRET_VALUE>" \| gcloud secrets versions add "manual-ios-bundle-id" --data-file=-`
`manual-ios-certificate`	The value of the Base64 converted `.p12` file that you will obtain during iOS configuration	Set now if you know what this value will be - otherwise create a placeholder and update after completing your iOS deployment (leave as placeholder if you will be deploying to Android only)	`echo -n "<SECRET_VALUE>" \| gcloud secrets versions add "manual-ios-certificate" --data-file=-`
`manual-ios-certificate-password`	The value of the password for the `.p12` certificate (necessary if your certificate is encrypted - otherwise leave empty)	Set now if you know what this value will be - otherwise create a placeholder and update after completing your iOS deployment (leave as placeholder if you will be deploying to Android only)	`echo -n "<SECRET_VALUE>" \| gcloud secrets versions add "manual-ios-certificate-password" --data-file=-`
`manual-ios-deeplink-url`	The sign-in screen is run on the Hydra-based auth server. This URL is a deep link that helps redirect users to the native mobile app after they sign in. (for example, `app://{PREFIX}-{ENV}.{DOMAIN}/mystudies`)	Set now if you know what this value will be - otherwise create a placeholder and update after completing your iOS deployment (leave as placeholder if you will be deploying to Android only)	`echo -n "<SECRET_VALUE>" \| gcloud secrets versions add "manual-ios-deeplink-url" --data-file=-`

Note: When updating secrets after this initial deployment, you must refresh your Kubernetes cluster and restart the relevant pods to ensure the updated secrets are propagated to your applications (you do not need to do this now - only when making updates later), for example you can update your Kubernetes state with:

cd $GIT_ROOT/deployment/terraform/kubernetes
terraform init && terraform apply

then, restart the pods by deleting them in the Kubernetes dashboard or running:

APP_PATH=<path_to_component_to_restart> # for example, $GIT_ROOT/auth-server
kubectl scale --replicas=0 -f $APP_PATH/tf-deployment.yaml && \
kubectl scale --replicas=1 -f $APP_PATH/tf-deployment.yaml

If you rotate an application’s ‘client_id’ or ‘client_secret’, such as auto-response-datastore-client-id or auto-response-datastore-secret-key, you must register the new values in Hydra by re-running the register_clients_in_hydra.sh script or executing the appropriate REST requests directly (see /hydra/README.md for more information about working with Hydra manually)

Finish Kubernetes cluster configuration and deployment

Configure the remaining resources with Terraform, for example:

cd $GIT_ROOT/deployment/terraform/kubernetes/
terraform init && terraform apply

Set your kubectl credentials, for example:

gcloud container clusters get-credentials "$PREFIX-$ENV-gke-cluster" \
  --region=$LOCATION --project="$PREFIX-$ENV-apps"

Apply the pod security policies, for example:

kubectl apply \
  -f $GIT_ROOT/deployment/kubernetes/pod_security_policy.yaml \
  -f $GIT_ROOT/deployment/kubernetes/pod_security_policy-istio.yaml

Apply all deployments, for example:

kubectl apply \
  -f $GIT_ROOT/study-datastore/tf-deployment.yaml \
  -f $GIT_ROOT/response-datastore/tf-deployment.yaml \
  -f $GIT_ROOT/participant-datastore/consent-mgmt-module/tf-deployment.yaml \
  -f $GIT_ROOT/participant-datastore/enroll-mgmt-module/tf-deployment.yaml \
  -f $GIT_ROOT/participant-datastore/user-mgmt-module/tf-deployment.yaml \
  -f $GIT_ROOT/study-builder/tf-deployment.yaml \
  -f $GIT_ROOT/auth-server/tf-deployment.yaml \
  -f $GIT_ROOT/participant-manager-datastore/tf-deployment.yaml \
  -f $GIT_ROOT/hydra/tf-deployment.yaml \
  -f $GIT_ROOT/participant-manager/tf-deployment.yaml

Apply all services, for example:

kubectl apply \
  -f $GIT_ROOT/study-datastore/tf-service.yaml \
  -f $GIT_ROOT/response-datastore/tf-service.yaml \
  -f $GIT_ROOT/participant-datastore/consent-mgmt-module/tf-service.yaml \
  -f $GIT_ROOT/participant-datastore/enroll-mgmt-module/tf-service.yaml \
  -f $GIT_ROOT/participant-datastore/user-mgmt-module/tf-service.yaml \
  -f $GIT_ROOT/study-builder/tf-service.yaml \
  -f $GIT_ROOT/auth-server/tf-service.yaml \
  -f $GIT_ROOT/participant-manager-datastore/tf-service.yaml \
  -f $GIT_ROOT/hydra/tf-service.yaml \
  -f $GIT_ROOT/participant-manager/tf-service.yaml

Apply the certificate and the ingress, for example:

kubectl apply \
  -f $GIT_ROOT/deployment/kubernetes/cert.yaml \
  -f $GIT_ROOT/deployment/kubernetes/ingress.yaml

Update firewalls:
- Run kubectl describe ingress $PREFIX-$ENV
- Look at the suggested commands under "Events", in the form of "Firewall change required by network admin"
- Run each of the suggested commands

Verify the status of your Kubernetes cluster
- Check the Kubernetes ingress dashboard in your {PREFIX}-{ENV}-apps project to view the status of your cluster ingress (if status is not green, repeat the firewall step above)
- Check the Kubernetes workloads dashboard in your {PREFIX}-{ENV}-apps project to view the status of your applications (confirm all applications are green before proceeding - it can take up to 15 minutes for all containers to become operational)
- Check the status of your Kubernetes cluster SSL certificates on the certificates page of your {PREFIX}-{ENV}-apps project’s load balancing ‘advanced menu’ (both the participants.{PREFIX}-{ENV}.{DOMAIN} and studies.{PREFIX}-{ENV}.{DOMAIN} certificates must be green for your deployment to use https)
Configure your initial application credentials
- Create the Hydra credentials for server-to-server requests by running register_clients_in_hydra.sh, for example:
```
$GIT_ROOT/deployment/scripts/register_clients_in_hydra.sh \
  $PREFIX $ENV $DOMAIN
```
- Create your first admin user account for the Participant manager application by running the create_participant_manager_superadmin.sh script to generate and import a SQL dump file for the Participant datastore database (the password you specify must be at least 8 characters long and contain lower case, upper case, numeric and special characters), for example:
```
$GIT_ROOT/deployment/scripts/create_participant_manager_superadmin.sh \
  $PREFIX $ENV <YOUR_DESIRED_LOGIN_EMAIL> <YOUR_DESIRED_PASSWORD>
```
- Create your first admin user account for the Study builder application by running the create_study_builder_superadmin.sh script to generate and import a SQL dump file for the Study datastore database, for example:
```
sudo apt-get install apache2-utils -y
$GIT_ROOT/deployment/scripts/create_study_builder_superadmin.sh \
  $PREFIX $ENV <YOUR_DESIRED_LOGIN_EMAIL> <YOUR_DESIRED_PASSWORD>
```

Configure your first study

Navigate your browser to studies.{PREFIX}-{ENV}.{DOMAIN}/studybuilder/ (the trailing slash is necessary) and use the account credentials that you created with the create_study_builder_superadmin.sh script to log into the Study builder user interface
Change your password, then create any additional administrative accounts that you might need
Create a new study with the App ID that you set in the manual-mobile-app-appid secret, or choose a new App ID that you will update manual-mobile-app-appid with
Publish your study to propagate your study values to the other platform components
Navigate your browser to participants.{PREFIX}-{ENV}.{DOMAIN}/participant-manager/ (the trailing slash is necessary), then use the account credentials that you created with the create_participant_manager_superadmin.sh script to log into the Participant manager user interface (if the Participant Manager application fails to load, confirm you are using https - this deployment requires https to be fully operational)
You will be asked to change your password; afterwards you can create any additional administrative accounts that you might need
Confirm your new study is visible in the Participant manager interface

Prepare your mobile applications

Follow the instructions in either or both Android and iOS deployment guides (if you haven’t created a study yet, you can configure the mobile applications with the APP_ID you plan on using when you create your first study in the Study builder)
Open Secret Manager in your {PREFIX}-{ENV}-secrets project and update the secrets you previously configured with placeholder values (you can skip this step if you already configured your secrets with the appropriate values - if you do update secret values, make sure to refresh your Kubernetes cluster and applications as described above)
- manual-mobile-app-appid is the value of the App ID (15 characters max) that you configured, or will configure, on the Settings page of the Study builder
- manual-android-bundle-id is the value of applicationId that you configured in Android/app/build.gradle, for example {PREFIX}_{ENV}.{DOMAIN} (note that some characters are not permitted)
- manual-fcm-api-url is the URL of your Firebase Cloud Messaging API
- manual-android-server-key is your Firebase Cloud Messaging server key
- manual-android-deeplink-url is the URL to redirect to after Android login (for example, app://{PREFIX}-{ENV}.{DOMAIN}/mystudies)
- manual-ios-bundle-id is the value you obtained from Xcode (in production use-cases, this bundle ID needs to be verified with Apple)
- manual-ios-certificate is the value of the Base64 converted .p12 file
- manual-ios-certificate-password is the value of the password for the .p12 certificate
- manual-ios-deeplink-url is the URL to redirect to after iOS login
Initialize your Participant datastore database to work with your mobile applications
- Once a study with the App ID corresponding to the manual-mobile-app-appid secret is created and published using the Study builder user interface, a corresponding app record will appear in the Participant manager user interface (if you created a study before all of your platform components were operational, you can reinitialize this process by using the Study builder user interface to pause and resume the study)
- Once the App ID appears in Participant manager, run the deployment/scripts/copy_app_info_to_sql.sh script to update the remaining databases, for example:
```
$GIT_ROOT/deployment/scripts/copy_app_info_to_sql.sh $PREFIX $ENV
```

Clean up

Remove your user account from the groups you no longer need access to

Revoke user access in your environment, for example:

gcloud auth revoke <user>@<domain> -q && \
  gcloud auth application-default revoke -q

Optionally, confirm no other users are logged in, for example:
```
gcloud auth list
```

Maintaining & Updating FDA MyStudies

Database Migration

When updating FDA MyStudies it may be necessary to migrate the databases to support the new version. Detailed instructions can be found in the Database Migration README.

Study Resources in Cloud Storage (2.0.5 upgrade)

Release 2.0.5 changed permissions for the Cloud Storage buckets used for study resources to remove public access and use signed URLs. New deployments will use this behavior automatically. When upgrading a prior release to 2.0.5 or greater, you will need to change permissions to the storage bucket using one of the following processes.

Manual process:

Go to Data Project ({prefix}-{env}-data) and remove AllUser access from the ({prefix}-{env}-mystudies-study-resources) storage bucket and provide access to the service accounts below with storage.objectAdmin role

serviceAccount:participant-manager-gke-sa@{{.prefix}}-{{.env}}-apps.iam.gserviceaccount.com
serviceAccount:study-builder-gke-sa@{{.prefix}}-{{.env}}-apps.iam.gserviceaccount.com
serviceAccount:study-datastore-gke-sa@{{.prefix}}-{{.env}}-apps.iam.gserviceaccount.com

Script process:

In the Terraform/{prefix}-{env}-data /main.tf file, please replace the existing module "{prefix}_{env}_mystudies_study_resources>" with the values below, replacing with your prefix and env values

module "{prefix}_{env}_mystudies_study_resources" {
  source = "terraform-google-modules/cloud-storage/google//modules/simple_bucket"
  version = "~> 1.4"

  name = "{prefix}-{env}-mystudies-study-resources"
  project_id = module.project.project_id
  location = "us-east1"

  iam_members = [
    {
      member = "serviceAccount:study-builder-gke-sa@{prefix}-{env}-apps.iam.gserviceaccount.com"
      role = "roles/storage.objectAdmin"
    },
    {
      member = "serviceAccount:study-datastore-gke-sa@{prefix}-{env}-apps.iam.gserviceaccount.com"
      role = "roles/storage.objectAdmin"
    },
    {
      member = "serviceAccount:participant-manager-gke-sa@{prefix}-{env}-apps.iam.gserviceaccount.com"
      role = "roles/storage.objectAdmin"
    },
  ]
}

tfengine process:

Pull the latest code (2.0.5+) and run the following commands

cd $GIT_ROOT
tfengine --config_path=$ENGINE_CONFIG --output_path=$GIT_ROOT/deployment/terraform
git checkout -b bucket-permissions
git add $GIT_ROOT/deployment/terraform
git commit -m "bucket-permissions"
git push origin bucket-permissions

Then create a pull request from bucket-permissions branch to your target branch Once your pull request pre-submit checks have completed successfully, and you have received code review approval, merge your pull request to trigger terraform apply(you can view the status of the operation in the Cloud Build history of your devops project)

Study Import / Export (2.0.6 upgrade)

Release 2.0.6 added additional functionality to support study import and export. This requires a permission change and an additional secret to be added. When upgrading a prior release to 2.0.6 or greater, you will need to perform the following steps.

Permission change to import bucket

Update your repository with the latest changes from release 2.0.6 or greater, create a new working branch and make the following changes:

In the file deployment/terraform/{prefix}-{env}-data/main.tf find the section module "{prefix}_{env}_mystudies_sql_import" { [...] } and completely replace it with the following, substituting your values for {prefix} & {env} and changing the location to your preference:

module "{prefix}-{env}_mystudies_sql_import" {
source  = "terraform-google-modules/cloud-storage/google//modules/simple_bucket"
version = "~> 1.4"

name       = "{prefix}-{env}-mystudies-sql-import"
project_id = module.project.project_id
location   = "us-east1"

iam_members = [
  {
    member = "serviceAccount:${module.mystudies.instance_service_account_email_address}"
    role   = "roles/storage.objectViewer"
  },
  {
    member = "serviceAccount:study-builder-gke-sa@{prefix}-{env}-apps.iam.gserviceaccount.com"
    role   = "roles/storage.objectAdmin"
  },
]
}

Create a pull request from this working branch to your specified branch, which will start the terraform plan and validation. After completion of the plan and validation, merge the pull request. That will run the terraform apply.

Add import / export bucket to Kubernetes cluster shared secrets

To add the bucket to the shared secrets, create a new working branch and make the following change:

Edit the file deployment/terraform/kubernetes/main.tf and in the section # Shared secrets add the following line to the section data = { [...] }, substituting your values for {prefix} & {env}
```
study_export_import_bucket_name   = "{prefix}_{env}-mystudies-sql-import" 
```
Create a pull request from this working branch to your specified branch, which will start the terraform plan and validation. After completion of the plan and validation, merge the pull request. That will run the terraform apply.
Pull the latest code from your repository and checkout your specified branch which contains the new shared secret.

Run the following commands to apply the changes to your cluster:

cd $GIT_ROOT/deployment/terraform/kubernetes/
terraform init && terraform apply

Run the following command to apply the latest Study Builder deployment changes:
```
kubectl apply \
  -f $GIT_ROOT/study-builder/tf-deployment.yaml
```

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

deployment

deployment

README.md

Deploying FDA MyStudies

Introduction

Deployment overview

Figure 1: Overall architecture of the semi-automated deployment

Figure 2: Application architecture

Before you begin

Prepare the cloud platform

Set up your environment

Create your devops project and configure CICD pipelines

Deploy your platform infrastructure

Configure your domain for the deployment

Configure your deployment’s databases

Configure and deploy your applications

Configure your first study

Prepare your mobile applications

Clean up

Maintaining & Updating FDA MyStudies

Database Migration

Study Resources in Cloud Storage (2.0.5 upgrade)

Manual process:

Script process:

tfengine process:

Study Import / Export (2.0.6 upgrade)

Permission change to import bucket

Add import / export bucket to Kubernetes cluster shared secrets

Files

deployment

Directory actions

More options

Directory actions

More options

Latest commit

History

deployment

Folders and files

parent directory

README.md

Deploying FDA MyStudies

Introduction

Deployment overview

Figure 1: Overall architecture of the semi-automated deployment

Figure 2: Application architecture

Before you begin

Prepare the cloud platform

Set up your environment

Create your devops project and configure CICD pipelines

Deploy your platform infrastructure

Configure your domain for the deployment

Configure your deployment’s databases

Configure and deploy your applications

Configure your first study

Prepare your mobile applications

Clean up

Maintaining & Updating FDA MyStudies

Database Migration

Study Resources in Cloud Storage (2.0.5 upgrade)

Manual process:

Script process:

tfengine process:

Study Import / Export (2.0.6 upgrade)

Permission change to import bucket

Add import / export bucket to Kubernetes cluster shared secrets