Deploy Airflow on Cloud Foundry using pip

[robert-bryson]

User Story

In order to [use Airflow], [datagov team] wants [to ensure that it can be deployed on the existing cloud foundry infrastructure].

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN [an Airflow repo]
  WHEN [a cf push] happens
  THEN [a successful deployment is made]

Background

[Any helpful contextual notes or links to artifacts/evidence, if needed]

• SPIKE: Create harvesting workflow using apache airflow. #4422

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

Sketch

• Attempt simple deployment to cloud.gov, using postgreSQL instance
• Attempt configuring 3 apps in cloud.gov: a UI, a timer, and a worker (which may have 2 instances).
• Move code to new https://github.com/GSA/datagov-harvester repo
• Configure Celery queue backend
• Confirm DAG able to complete using above infrastructure
• Confirm various features (debug, logs, workflow view, etc) are working and useful

If any of the above take longer than 3 days to complete, document blockers and status and move on where we can.

Potential additional work;

• wire up github actions
• experiment with dockerized deploys

[robert-bryson]

!!!

[robert-bryson]

Where we are on this:

• Docker/pip deploys both work if needed, but using python buildpack is probably best option
• Airflow standalone can be deployed with with manifest-standalone.yml and is suitable for development/testing.
• Airflow multi is mostly there, but a number of issues remain: setting up celery/flower, logging issue, need to figure out auth, etc.

[robert-bryson]

I think for the first time I can get everything to be up via a cf push:

And even mostly healthy:

Although the CeleryExecutor isn't yet wired up correctly:

[robert-bryson]

Well, it ain't much but I did finally get the dag-processor to show healthy:

[robert-bryson]

This spike is wildly overdue so I will close it at the current state and write up additional tickets for the remaining issues with the deployment. Overall a proof of concept for deploying Airflow on cloud.gov was successful though with challenges. I think with the feeling that we may explore other deployment options (k8s, rabbitmq, etc) this work should be paused.

Done:

• Airflow standalone on cf
• Airflow standalone docker on cf
• Airflow multi-tenant deployment on cf
  • Working webserver/metastore/scheduler/dag processor/etc

Remains:

• Issues around CeleryExecutor and redis (Airflow on CF: CeleryExecutor/redis communication issues #4471)
• Bug with s3 authentication for storing logs (Configure S3 to store Airflow Logs #4470)
• Any Github Actions CI/CD