Best Practices for sharing modular pipelines

[broehrig]

Hi everyone,
we have a few steps that all our kedro pipelines use, just with different parameters. We are thinking about using modular pipelines to implement this in a reusable way.

However, I am unsure how to do the actual sharing: Using kedro pipeline pull copies the source code into the project. I am afraid that this would lead to the different usages of the modular pipeline diverging over time. Since the pipeline is reused in many different repositories, that seems like a big risk to me.

On the other hand, actually sharing the code in this way would enable each project that reuses the pipeline to make small adjustments that might be needed for specific use cases. If we go for an importable library rather than copying the code, we might run into the issue that the shared pipeline grows with conditionals satisfying all the possible use cases.

So I would really like to hear your opinion on this matter - How do you deal with the issue of sharing code / modular pipelines between kedro projects? What works and doesn't work for you?

[datajoely]

I'm going to paste what I wrote last night on Discord:

This is a really interesting topic of discussion and in truth, something where best practice is still being defined. I would say that this is somewhere that feels similar to API versioning where a level of governances is needed, breaking changes need to be communicated and transparent etc.
I have some initial thoughts:

1. The smaller the modular pipeline the easier it is to manage changes. Several of our internal use-cases have very simple pipelines e.g. Splitting for cross validation is reused across several pipelines and projects.
2. This also brings in questions related to Governance, mono-repo vs multi-repo where things are centralised versus decentralised. > 3. We currently allow users to pull as source code, rather than importable and installable libraries as we want be people to be able to remix and modify pipelines to fit their needs.

Questions for the community:

1. What is working for them today?
2. What isn't
3. What would you love to see added in the future?

[idanov]

@broehrig It's a tradeoff as you described it. Packaging modular pipelines and using them as normal pip-installable python packages is the best case, but then you run into the problem of not being able to slightly modify the code to fit your usecase. On the other hand pulling into your source code gives you that flexibility at the expense of harder updates of those pipelines.

Some projects approach the problem by creating "versions" of the pipeline, e.g. in your pipeline package you have multiple subpackages named v1, v2, v3 etc. Not sure if that makes things easier to manage, especially when you need to modify source code for some of the nodes.

Another approach I'd suggest is to ensure that your pipelines are made up from trully modular nodes. Then the modification of the pipeline happens through a node replacement. You pull your pipeline and never change the actual code in it (so to preserve ability to upgrade the code), but you only modify the pipeline creation part by using the one provided from the package and then removing the node you want to modify and adding a new replica with the modified code.

None of those options are perfect, but code sharing with allowing modifications is quite hard to do in any case and is fundamentally limited by the same problems any distributed system faces.

[tomaz-suller]

I think before I mention my ideas, it's good to note I have never shared Kedro pipelines with other people. I've just thought about how I'd do it.

I believe distributing ready-made pipelines in packaged modules wouldn't work well precisely because of what @datajoely mentioned - more often than not you have to adapt the external pipeline to work with your data or to just add functionality to it.

Considering then the best way to distribute these would be through source code, a simple idea would be to just have each of these shared pipelines as a Git submodule or something like that. People would be able to clone it into their pipelines folder and adapt as needed; plus, updates to the pipeline could be incorporated by pulling the submodules. I see two disadvantages to that though:

1. Creating a huge number of repositories for different pipelines; it could be an issue for larger organisations and such, but I think it wouldn't be that bad for the average team with up to say 4 shared pipelines;
2. Unifying all pipeline configs into a single parameters.yaml and/or catalog.yaml.

The second issue I think is the hardest one to tackle. A quick google search suggested some tools for dealing with YAML files do support include statements, which could make the process easier. There could also be some kedro submodule add CLI functionality that extended Git's submodule to deal with the config files as well to facilitate things.

So overall, I think Git submodules would offer a good solution to sharing pipelines, however creating the issues of managing a great number of repositories (one per pipeline most likely) and of config files, or unifying the configs somehow.

[datajoely]

@tomaz-suller this is a great time to also share the 'micro-packaging' work we've been focusing on during in the couple months since this conversation cropped up.

@lorenabalan recently presented the work-in-progress demo at PackagingCon. The spliced video hasn't been posted so I've linked to the timestamp. I'm pretty sure the workflow Lorena demonstrates is what you've described.

The configuration overhead is another issue we're also tracking on #891

[tomaz-suller]

@datajoely I do agree it's quite similar and to be honest much broader in scope than what I considered. Frankly I think it's a great way to do it.

The only unaddressed point I can think of is versioning. For instance, imagine a team is working on constructing a new shared pipeline all together; then pulling new versions of the pipeline or scripts after every modification seems less than ideal. Even if you are not a developer and are just a user of a pipeline, it could be the case the pipeline's updated and you lag behind. That's why I mentioned git submodules or something similar.

I reckon you guys know a lot more about packaging and distribution than I do, so might know of a way to distribute the .tar.gz packages automatically as well, for instance generating them in some CI pipeline and then pushing to a remote location (maybe even cloud storage?) so everyone can just pull from there and get the update.

[datajoely]

@tomaz-suller - you make some very good points. In truth we're treading in somewhat unknown waters here - so we're learning as we go a long. This sort of syncing overhead is something we're going to have to track and quantify once the functionality is in the hands of users.

So in short it's not something I think we should try and fully anticipate just yet - but listen to users once they use it in anger.