Zerops x Apache Airflow

Apache Airflow is an open-source platform for orchestrating and scheduling complex data pipelines. It allows you to programmatically author, schedule, and monitor workflows as Directed Acyclic Graphs (DAGs).

Deploy on Zerops

Manually copy the import yaml to the import dialog in the Zerops app.

project:
  name: Apache Airflow Standalone
  tags:
    - zerops-recipe
    - development
services:
  - hostname: airflowstandalone
    type: [email protected]
    verticalAutoscaling:
      minRam: 1
    maxContainers: 1
    buildFromGit: https://github.com/zeropsio/recipe-airflow
    enableSubdomainAccess: true

For production-ready, high-available setup use production import yaml.

Recipe Features

• Apache Airflow version 2.10
• Python version 3.12
• Showcase DAG file

Development (Standalone)

• Standalone mode (docs), not recommended for production usage
• Access details can be found in runtime log, look for logs as the following:

standalone | Airflow is ready
standalone | Login with username: admin  password: *****
standalone | Airflow Standalone is for development purposes only. Do not use this in production!

Production

Warning

Airflow production setting is currently only experimental, since the shared storage for sharing DAG files is not working properly.

• Data are stored in Postgres
• Celery executor (running on redis)
• DAG files are distributed via shared storage (shared mounted volume)
• Only push to the airflowdags service to update your DAG files (either via zcli push airflowdags or connect your Git with the airflowdags service), other services will use the files located in the mounted volume
• UI access:
  • Username: admin
  • Password: $ADMIN_PASSWORD generated project environment variable (visible in GUI)

Ways To Reduce Cost

• Change worker count
• Decrease scaling resources to minimum
• Change database services mode to NON_HA (historical data can be lost)

Running Your Own DAGs

To use your own data pipelines, simply:

1. Fork this repository
2. Deploy the project (or the individual services to an existing project) to Zerops
3. Replace the insides of dags folder with your own pipelines
4. Push to airflowdags service via zCLI or connect the airflowdags with the forked repository

Need help setting your project up? Join Zerops Discord community.