Run DAG on schedule unless it has been run since last scheduled run

[chrishoage]

One thing I have not been able to figure out is how to prevent a DAG from running if it has been run before the scheduler would typically run the job.

I'm wondering if there is a cleaner way than to use the HTTP executor to get the finishedAt attribute, get the current time from shell, parse the cron string, and do the math to see if it's run since the last run?

Some questions:

• are any of the DAG details exposed via the environment available to the DAG?
• is there a DAG setting available to achieve this behavior that I missed in the documentation?
• aside from creating a collection of scripts and exposing them on the PATH is there a way to share "DAG functions" between DAGs?
• is there any environment variables set for scripts or commands that dagu invokes that would let the script know which DAG is invoking it?

I would like to share this "delay run if run before scheduled run" between DAGs, and it seems like it's going to be a little cumbersome.

I believe the script will need to:

• be told what DAG is executing it directly in a step
• fetch the detail status of the DAG
• read the DAG file to determine the schedule
• parse the cron line into a schedule
• compare the last finish with the next scheduled run if the DAG is not in a running state
• use the output of this command as a precondition for continuing the DAG

Is there a simpler way?

[yohamta]

Hey Chris,

Thanks for the detailed questions! You've brought up some really good points about how we can make DAG management in Dagu even better.

Preventing a DAG from rerunning if it has already completed is a smart idea. I'm thinking of adding a runOnlyOnSchedule flag at the DAG level. If you set that to true, Dagu will automatically check the last successful run time against the schedule and skip the current run if it's not needed. This should make your current workaround unnecessary.

Regarding your other questions:

• DAG Details: We actually already have some special environment variables documented here. We could definitely add more, like DAG_SCHEDULE and DAG_LAST_SUCCESSFUL_RUN, to give your tasks more info. What do you think?

• Sharing DAG Functions: Right now, we support sub-DAGs (check out the docs here). We know it's not quite the same as reusable functions, but it might help in some cases. We're definitely thinking about better ways to modularize DAGs, so your input is super helpful. One limitation right now is that a single DAG can't run more than once concurrently, but I'm planning to change that.

• Knowing Which DAG is Running: The DAG_REQUEST_ID environment variable is available during execution, which can be useful for tracking.

That runOnlyOnSchedule flag seems particularly helpful, so I'm definitely going to look into implementing that. Thanks again for the great feedback! Let us know what you think.

[chrishoage]

We actually already have some special environment variables documented here. We could definitely add more, like DAG_SCHEDULE and DAG_LAST_SUCCESSFUL_RUN, to give your tasks more info. What do you think?

I've found these which lead me to wonder if there may be more undocumented ones.

I do wonder however since jq support is a first class executor if it would be simpler to simply expose the details as DAG_DETAILS_JSON

Then steps could be written like this

steps:
  - name: get value
    executor: jq
    command: '{( .schedule }}'
    script: "echo $DAG_DETAILS_JSON
   output: DAG_SCHEDULE

That isn't to say having bespoke environment variables wouldn't be handy for more frequently accessed values. It would save needing a step simply to fetch the details.

we support sub-DAGs (check out the docs here). We know it's not quite the same as reusable functions, but it might help in some cases.

sub-DAGs are one of the primary ways I am planning on using this tool.

I have a media ingestion script that is currently a monolith, but there's no reason why it could not be split up so parts of this script could be run on demand. I plan to break the major parts of this script into DAGs and have a meta-DAG that runs them together on a schedule.

Knowing Which DAG is Running: The DAG_REQUEST_ID environment variable is available during execution, which can be useful for tracking.

There doesn't appear to be a straight forward way from the API to know what DAG is running from the request id alone. It seems it would require hitting GET /api/v1/dags for the list of DAGs the searching this list for the request ID.

We would need DAG_NAME in order to easily look up the detail of a DAG from a script via the API

That runOnlyOnSchedule flag seems particularly helpful, so I'm definitely going to look into implementing that.

Thank you! To add more context as to why runOnlyOnSchedule would be useful:

With my media ingestion script mentioned above at the end I run a backup command. This backup command is expensive computationally, and also wakes up spinning hard drives that I would prefer to keep spun down if possible.

My media ingestion script doesn't run every day however, and if there is nothing to ingest I don't wish to run the backup command.

However I would still like to run the backup command periodically.

My goal is to have the backup command in a DAG on a schedule, and have my meta-DAG run this backup command DAG. However the next time the schedule triggers I don't want the backup DAG to run if it ran since the last time it was supposed to run on schedule.

For example:

• backup DAG is scheduled to run every 3 days.
• backup DAG is triggered manually 1.5 days after its normally scheduled run
• backup DAG schedule timer fires, but it has already run recently, so it doesn't run.
• 3 days pass
• backup DAG schedule timer fires, it has not run in the last 3 days, so it runs.

Thanks again for the great feedback! Let us know what you think.

Thank you for this software! I'm still evaluating it but looks like it is going to fit my needs. I have further ideas, but I'll wait to supply these until I have done more than read the docs and the code 😅