How to reuse or reloop kedro pipelines in an unique run.

[vinialbert]

Hello, everyone!

Kedro is a very special tool for data science. In this sense, I'd like to know if there is a way to re-run or re-loop pipeline or parts of a pipeline during a single execution. It would be very helpful to train multiple models with different input data. Did you guys know how to deal with this? Also, I'm curious if there is a way to use partitioned dataset to do this. For instance, if one partition pass through all the pipeline before a second partition starts.

[datajoely]

Hi @vinialbert - you can use modular pipelines to reuse the same pipeline structure with different inputs.

I'm currently working on a proper example project, which is very much work in progress - but may be helpful.

I wouldn't go down the partitioned dataset route I would use the knowledge that my pipeline(s) are static in structure but can be parameterized with different inputs.