📖 Refactor Data-Engineering-Exports to Terraform

[jhpyke]

User Story

As an ops engineer
I want to refactor data-engineering-exports to use terraform
So that I can maintain it without learning pulumi.

Value / Purpose

We currently deploy the assets in data-engineering-exports using pulumi. However, attempts to automate this have been frustrated by unclear deprecation warnings with no apparent source. As such, it would be easier long term to move the assets managed into a terraform deployment, such that we can automate the process, and all team members can understand and service the tool.

Useful Contacts

@jhpyke

User Types

AP Team

Hypothesis

If we can automate this in pulumi, then we can reduce the mental bandwidth required to run the tool, as we will no longer have to service ad-hoc requests for state updates from users.

Proposal

The project deploys two sets of assets for users - the pull buckets (individual buckets per user project, with specific IAM permissions for them), and push bucket (a shared bucket, with policies for individual users providing write access. These are accompanied lambdas that scan for new files and fire them to a specified target bucket). Each of these represents a couple of simple, repeatable structures that should be easily representable as terraform infra.

Additional Information

All infrastructure is deployed into the data-production account, but the backend pulumi state is managed from the data-engineering account. This shouldn't matter but noting in case it does.

Definition of Done

• Replicate pull bucket functionality in terraform
• Import existing pull buckets to terraform
• Replicate push bucket funcationality in terraform
• Import existing push bucket and lambdas to terraform

[github-actions]

This issue is being marked as stale because it has been open for 60 days with no activity. Remove stale label or comment to keep the issue open.

[github-actions]

This issue is being closed because it has been open for a further 7 days with no activity. If this is still a valid issue, please reopen it, Thank you!