Dynamic DAG

[alexander-myltsev]

Hi,

Thank you for the framework. I'm on 0.16.6.

I noticed from the source code that Kedro DAG is very static. That causes some questions.

1. I have a data source:

• it is an SQL database that returns names of CSV files by some DS_ID
• the next part is to fetch CSV files with data. The count varies.

I'd like to parametrize DS_ID at catalog.yml per each run at production. The only possibility I found is as follows:

• register TemplatedConfigLoader at hooks
• create conf/concrete_ds_id/globals.yml folder on the flight and pass --env concrete_ds_id to run.

Is there a better way?

2. Next thing. I'd like to compose a pipeline based on fetched CSV files. I'd like to see each of them as an input to a node. But I have no access to catalog items at ProjectHooks to create nodes dynamically at register_pipelines. How to do that?

[datajoely]

Hi @alexander-myltsev - dynamic configuration is something we're looking to address more robustly in the future, but the mechanisms you describe are close to what I would recommend today. TempaltedConfigLoader now has support for Jinja which may be helpful, although we encourage users to err on the side of simplicity here.

In general - we're keen to encourage as little 'magic' as possible. One of the trends we see emerging is that pipelines DAGs should be immutable (i.e. specific nodes and flows are static), but the configuration which defines the inputs/outputs can be dynamic. This is a subtle difference, but an important one in terms of readability/maintainability/mental model complexity etc.

Are you running Kedro inside an orchestrator (e.g. Airflow/Argo etc.)? Perhaps this is a situation where you can use the orchestrator to dynamically parametrise your runs via CLI commands / environment variables.