New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Determine how jobs will be linked #16

Open

Tracked by

franTarkenton opened this issue Jul 10, 2023 · 0 comments

Open

Tracked by

Determine how jobs will be linked #16

franTarkenton opened this issue Jul 10, 2023 · 0 comments

Member

franTarkenton commented Jul 10, 2023

There will be 4 separate jobs that need to be run before the R based data ingestion script can be run.

Need to determine how to deal with convergent jobs.

Some options:

Implement individual jobs as DAGS in airflow pipeline and the the R based data get triggered when the 4 data prep jobs are complete
Evaluate if a trigger can be setup using GHA that will only progress if all 4 jobs are complete.
Staggered cron jobs (possibly as initial option, but really want to avoid this option)
Other???

franTarkenton mentioned this issue

Climate Observations Pipeline #10

Open

10 tasks

franTarkenton added zen_reorg and removed zen_reorg labels

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment