Set up Cloud SQL Postgres database for dagster storage #2996

bendnorman · 2023-10-31T22:32:53Z

PR Overview

This PR sets up a Cloud SQL Postgres database for dagster storage in our nightly builds. This will hopefully prevent the dagster backend storage database from locking up and killing our builds.

Changes:

Creates a Cloud SQL Postgres console with 2 CPUs and 8 GB of RAM. @jdangerx and I decided handling the instance lifecycle from within the container might cause some jankyness so we will keep the instance running 24/7 which will cost about $101 a month. We will explore running postgres within our VM container to save cost and the overhead of managing an external db Spin up a postgres database within our nightly build container #3003.
Adds logic in docker/gcp_pudl_etl.sh to start and stop the dagster-storage Cloud SQL instance and create and delete a dagster-storage-$ACTION_SHA-$GITHUB_REF database. This way, we have a fresh database for each nightly build run and won't incur Cloud SQL instance charges when we aren't running a nightly build.
Add static external IP addresses to the pudl-deployment-tag and pudl-deployment-dev VMs so they can be whitelisted by the Cloud SQL instance.
Points the dagster deployment to the Postgres database
Passes the PUDL_SETTINGS_YAML to the container. I think this var was set manually in the GCP Console.

PR Checklist

Merge the most recent version of the branch you are merging into (probably dev).
All CI checks are passing. Run tests locally to debug failures
Make sure you've included good docstrings.
For major data coverage & analysis changes, run data validation tests
Include unit tests for new functions and classes.
Defensive data quality/sanity checks in analyses & data processing functions.
Update the release notes and reference reference the PR and related issues.
Do your own explanatory review of the PR to help the reviewer understand what's going on and identify issues preemptively.

… up and down

…tabase name

codecov · 2023-11-01T04:29:48Z

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (b376141) 88.7% compared to head (c15e063) 88.7%.
Report is 78 commits behind head on dev.

Additional details and impacted files

@@          Coverage Diff          @@
##             dev   #2996   +/-   ##
=====================================
  Coverage   88.7%   88.7%           
=====================================
  Files         91      91           
  Lines      11007   11007           
=====================================
  Hits        9766    9766           
  Misses      1241    1241

Files	Coverage Δ
src/pudl/metadata/fields.py	`100.0% <ø> (ø)`
src/pudl/transform/ferc714.py	`100.0% <100.0%> (ø)`
src/pudl/transform/params/ferc1.py	`100.0% <ø> (ø)`
src/pudl/output/ferc1.py	`88.2% <69.2%> (ø)`
src/pudl/transform/ferc1.py	`96.7% <94.8%> (ø)`

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

jdangerx

This should work, I think, though lots of aspects of the postgres setup aren't actually in this PR.

I have some blocking questions about the lifecycle management stuff - I don't think we need to fully figure out how to make it robust, but we should at least talk about that before moving on.

jdangerx · 2023-11-01T14:06:05Z

.github/workflows/build-deploy-pudl.yml

@@ -85,6 +85,8 @@ jobs:

      # Deploy PUDL image to GCE
      - name: Deploy
+        env:
+          DAGSTER_PG_PASSWORD: ${{ secrets.DAGSTER_PG_PASSWORD }}


In the future we should probably store this secret in Google Secret Manager instead - but this is fine for now. We need to get our nightly builds working consistently again, so there's urgency. And, I think we're building up momentum to revamp the infra completely - what's one more piece of jankiness at this juncture?

You're saying pass secrets from Google Secret Manager directly to the VM instead of storing GCP secrets in Github Actions and passing them to the VM?

jdangerx · 2023-11-01T14:11:06Z

.github/workflows/build-deploy-pudl.yml

@@ -110,6 +112,11 @@ jobs:
            --container-env AWS_ACCESS_KEY_ID=${{ secrets.AWS_ACCESS_KEY_ID }} \
            --container-env AWS_SECRET_ACCESS_KEY=${{ secrets.AWS_SECRET_ACCESS_KEY }} \
            --container-env AWS_DEFAULT_REGION=${{ secrets.AWS_DEFAULT_REGION }} \
+            --container-env DAGSTER_PG_USERNAME="postgres" \
+            --container-env DAGSTER_PG_PASSWORD="$DAGSTER_PG_PASSWORD" \
+            --container-env DAGSTER_PG_HOST="104.154.182.24" \


It might be nice to run Postgres in the container itself, so we don't have to do all the weird Cloud SQL lifecycle management stuff.

On the other hand, it is weird to run your application and database in the same container. But gcloud compute only supports one container per VM (unless you want to run docker compose manually on the VM). So I'm OK with the Cloud SQL stuff for now as well.

Based on my experience, it seems like it's best to keep the app and database containers separate. I went with Cloud SQL because it's what dagster recommends and it seemed simpler than figuring out how to launch multiple containers.

I guess it's a little strange to spin the database up and down but I don't want to pay for the time it goes unused.

jdangerx · 2023-11-01T14:12:17Z

docker/gcp_pudl_etl.sh

+    # Start dagster-storage Cloud SQL instance
+    gcloud sql instances patch dagster-storage --activation-policy=ALWAYS
+    # Create database
+    gcloud sql databases create $DAGSTER_PG_DB --instance=dagster-storage 


Does this wait for the database to spin up before returning?

Does create throw an error if the DB already exists? Do we expect the DB to persist data over runs, or do we want to drop everything between runs? I see we delete the database below, but what happens if we error out and get interrupted before we delete the database?

I'll double check the database is created before returning.

I figured it was fine to delete the database even if the build fails because the dagster event logs aren't actually that helpful when debugging build failures.

bendnorman · 2023-11-02T00:14:43Z

I removed the Cloud SQL lifecycle logic from the gcp_pudl_etl.sh script.

bendnorman added 6 commits October 31, 2023 14:28

Set up Cloud SQL postgres database for dagster storage

dfea5ef

Copy dagster.yaml after DAGSTER_HOME is created

05901ab

Add proper quoting rules to DAGSTER_PG_PASSWORD secret

02c52f6

Use max cpus for nightly builds and spin dagster-storage SQL instance…

dcfb8de

… up and down

Create and delete Cloud SQL db during nightly builds

ad8442d

Set PUDL_SETTINGS_YML to etl_full.yml and add git sha to Cloud SQL da…

ab9425a

…tabase name

bendnorman marked this pull request as ready for review November 1, 2023 01:49

bendnorman linked an issue Nov 1, 2023 that may be closed by this pull request

Resolve Event database is locked error in nightly builds #2977

Closed

bendnorman added 2 commits October 31, 2023 18:13

Add short github ref to database name

6592fad

Update DAGSTER_PG_DB with short git sha

2f85315

bendnorman requested a review from jdangerx November 1, 2023 07:05

jdangerx requested changes Nov 1, 2023

View reviewed changes

bendnorman mentioned this pull request Nov 1, 2023

Spin up a postgres database within our nightly build container #3003

Closed

bendnorman marked this pull request as draft November 1, 2023 23:55

bendnorman added 2 commits November 1, 2023 15:55

Merge branch 'dev' into setup-dagster-postgres

1c6dd24

Remove Cloud SQL lifecycle management from gcp_pudl_etl.sh script

5b33500

Merge branch 'dev' into setup-dagster-postgres

c15e063

bendnorman marked this pull request as ready for review November 2, 2023 18:47

jdangerx approved these changes Nov 2, 2023

View reviewed changes

bendnorman merged commit 18bb60f into dev Nov 3, 2023

bendnorman deleted the setup-dagster-postgres branch November 3, 2023 00:01

bendnorman mentioned this pull request Nov 3, 2023

Add dagster postgres env vars to build-deploy-pudl.yaml #3014

Merged

8 tasks

zaneselvans mentioned this pull request Sep 26, 2024

Dagster SQLite EventLog failures #3868

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set up Cloud SQL Postgres database for dagster storage #2996

Set up Cloud SQL Postgres database for dagster storage #2996

bendnorman commented Oct 31, 2023 •

edited

Loading

codecov bot commented Nov 1, 2023 •

edited

Loading

jdangerx left a comment •

edited

Loading

jdangerx Nov 1, 2023

bendnorman Nov 1, 2023

jdangerx Nov 1, 2023

bendnorman Nov 1, 2023

jdangerx Nov 1, 2023

bendnorman Nov 1, 2023

bendnorman commented Nov 2, 2023

Set up Cloud SQL Postgres database for dagster storage #2996

Set up Cloud SQL Postgres database for dagster storage #2996

Conversation

bendnorman commented Oct 31, 2023 • edited Loading

PR Overview

PR Checklist

codecov bot commented Nov 1, 2023 • edited Loading

Codecov Report

jdangerx left a comment • edited Loading

Choose a reason for hiding this comment

jdangerx Nov 1, 2023

Choose a reason for hiding this comment

bendnorman Nov 1, 2023

Choose a reason for hiding this comment

jdangerx Nov 1, 2023

Choose a reason for hiding this comment

bendnorman Nov 1, 2023

Choose a reason for hiding this comment

jdangerx Nov 1, 2023

Choose a reason for hiding this comment

bendnorman Nov 1, 2023

Choose a reason for hiding this comment

bendnorman commented Nov 2, 2023

bendnorman commented Oct 31, 2023 •

edited

Loading

codecov bot commented Nov 1, 2023 •

edited

Loading

jdangerx left a comment •

edited

Loading