Add Built-in Execution Context Variables

[yohamta]

Overview
Introduce built-in execution context variables for each DAG execution.

Variables

• DAG_SCHEDULER_LOG_PATH
• DAG_EXECUTION_LOG_PATH
• DAG_REQUEST_ID

Example Usage

handlerOn:
  failure:
    command: "echo There was an error while doing ${DAG_REQUEST_ID} request"

Requirements

• Extend the DAGContext struct to hold necessary values
• Modify executors to pass required environment variables when running their commands
• Update documents (optional)

Additional Context

• [Feature] Built-in execution context variables #552

[halalala222]

@yohamta
hi , can i try to work on this issue ?

[yohamta]

Hi @halalala222 , thank you for your interest! That would be greatly helpful. Please don't hesitate to ask me know if you have any questions or need clarification on anything.

[halalala222]

@yohamta I have found some problems,when i work on this issue
In this issue requirements :

in /dag/scheduler/node.go

uitil.SplitCommandWithParse() function

the os.ExpandEnv() Will replace the string ${DAG_REQUEST_ID} with the corresponding environment variable.
so Modify executors to pass required environment variables when running their commands unable to implement the corresponding functionality.
Below are some methods I have considered for implementation:

1. When ${val} starts with 'dagu' or others indicating Built-in Execution Context Variables, skip the os.ExpandEnv(unescapeReplacer.Replace(v)). After that in executors can pass required environment variables when running their commands.
2. Use $(val) instead of ${val}, but it cannot rely on environment variables. Use strings.replaceAll to substitute.

I would like to ask if there is any better solution?

[halalala222]

@yohamta If the main scenario for this feature is to print the corresponding variables through a command execution, should other executors besides command be added?

[yohamta]

Hi @halalala222, thank you for the detailed explanation and those great solution ideas. This is very helpful. It is a problem that I was not aware of 😅 A possible solution that came to my mind is to replace the ${DAG_SCHEDULER_LOG_PATH} to some intermediate expression, such as {{DAG_SCHEDULER_LOG_PATH}} during the DAG building stage, and replace back to ${DAG_SCHEDULER_LOG_PATH} when node setup is called. What do you think?

[halalala222]

Hi @yohamta
I believe this is an excellent solution! I'll start working on it. Thank you! Your help has been invaluable.

[halalala222]

#654 Hi @yohamta , in this pr i try to work on it. I hope u can take a look at my PR. Thx!