Repurpose PSE terraform config to initialize cloud resources #2
Conversation
terraform/main.tf
Outdated
|
|
||
| variable "project_id" { | ||
| type = string | ||
| default = "catalyst-cooperative-pudl" |
There was a problem hiding this comment.
Is there anything weird about using the PUDL project from a different repo? Should we just create a mozilla project?
There was a problem hiding this comment.
If we're billing infrastructure costs to Mozilla, we should probably make a separate mozilla project - that will make the bookkeeping much easier.
If not, I think it's fine to use the same GCP project - though if we do that, I'd rather this terraform configuration live in the pudl repo. I think one tfstate per GCP project is easier to think about.
There was a problem hiding this comment.
The terraform stuff is a little funky. I personally think we should make a new Mozilla project in GCP, and set stuff up for that.
But if we're going to end up reusing a bunch of this infrastructure for PUDL one day, it might make sense to bring it into the main project + terraform state.
terraform/main.tf
Outdated
|
|
||
| variable "project_id" { | ||
| type = string | ||
| default = "catalyst-cooperative-pudl" |
There was a problem hiding this comment.
If we're billing infrastructure costs to Mozilla, we should probably make a separate mozilla project - that will make the bookkeeping much easier.
If not, I think it's fine to use the same GCP project - though if we do that, I'd rather this terraform configuration live in the pudl repo. I think one tfstate per GCP project is easier to think about.
terraform/main.tf
Outdated
There was a problem hiding this comment.
Might want to add @bendnorman too.
| } | ||
|
|
||
| resource "google_storage_bucket" "tfstate" { | ||
| name = "${random_id.bucket_prefix.hex}-tfstate" |
There was a problem hiding this comment.
So if we create a new Mozilla project, we'll need to actually follow this guide which basically says:
- use a block such as this one to create a bucket to hold tfstate
- only then do you configure the tfstate to be remote like in lines 1-4 - though point the gcs backend at the new bucket
If you want to stay in the PUDL project because the billing isn't separate, you can avoid making this new bucket and just use what we have already set up (the f344... bucket).
| type = "CLOUD_IAM_USER" | ||
| } | ||
|
|
||
| resource "google_secret_manager_secret" "postgres_pass" { |
There was a problem hiding this comment.
Note that you'll have to create a new "secret version" manually to actually hold a secret password value.
|
Ok so I've switched to using a dedicated mozilla project. We do have some funding for cloud resources, so we should be billing separately. It seems like that solves many of these problems. I think I need someone else to create the project though, as I don't have permission |
terraform/main.tf
Outdated
| @@ -0,0 +1,140 @@ | |||
| terraform { | |||
| backend "gcs" { | |||
| bucket = "f3441e415e6e5e7d-bucket-tfstate" | |||
There was a problem hiding this comment.
This is the right shape of things, but you'll run into issues with f34... bucket not existing in the new project! You'll have to:
- take away this
gcsbackend - create a new state bucket with a new random prefix
- re-configure this
gcsbackend with the new state bucket
It's fine to do that all locally before committing the changes, which should look exactly like this but with a different bucket name.
| display_name = "Mozilla dev" | ||
| } | ||
|
|
||
| resource "random_id" "bucket_prefix" { |
There was a problem hiding this comment.
super-nit: might be a little easier to read later if this bucket prefix is defined right next to the only place it's used - the tfstate bucket.
|
Oh and I think either @zaneselvans or @bendnorman have permissions to create a new project. |
|
|
||
| resource "google_project_iam_binding" "catalyst_people_editors" { | ||
| project = var.project_id | ||
| role = "roles/editor" |
There was a problem hiding this comment.
Is this making everyone an editor on the entire project?
This PR takes sets up cloud infrastructure using terraform. We will be using similar infrastructure to that used for the PSE project, so most of this was repurposed from there. Specifically, we will have storage bucket with raw 10-K filings, and postgres instance containing filing metadata. This will be used during development so we can quickly search through filings, manually inspect them, and do experimental development with minimal hassle. Eventually this infrastructure will likely be taken down, as all of the dev work for mozilla should be migrated into PUDL, and we will access filings using our normal archiver workflow.