Some Monorepo Questions

[idan-at]

Hi,

I am looking for a tool to manage a python monorepo. I've found this example, which suggests pdm can be used for that.
I wonder, can pdm be used to:

1. run tests on all changed projects (from the last git commit), and the (transitively) affected projects.
2. run lint on changed projects
3. publish packages independently (probably relying on git tags to determine the version)
4. share configurations on the root on the repository (flake configuration for example)

Thanks!

[idan-at]

@frostming Hey, WDYT?

This could be a game changer for Python monorepos.
I'd be happy to contribute if needed.

[majidaldo]

i did something like this where conda is a venv+pkg mgr https://github.com/pnnl/hydraconda/tree/master/project
so, b/c it's conda, you're not restricted to python too.

however, i wonder if i should just do nix and get better reproducibility.

[pawamoy]

Points 1, 2 and 4 sound more related to a task runner than a project manager IMO (which can of course be a task runner as well, so I understand the question).

What task runner do you currently use? Can't these 3 points be solved with it? Then integration within PDM (or any similar tool) is easy.

[pawamoy]

I don't think so, but I'm sure there are [Python] tools or libraries that are based on Git and provide such functionality (showing/working on the set of changed files in a given repository).

[idan-at]

It’s not just the files, you need to have an analysis based on the dependency tree

for example, if foo depends on bar, and bar code changes, then both foo and bar tests need to run

[pawamoy]

Yes, that is much more advanced. There's no such functionality in PDM. That would make a very interesting project though! Such a tool would need to read and understand pyproject.toml files to find all packages and build their local dependency graph, plus listing changed files using Git, combining both information to tell you which linter should run in which project.

I never worked with monorepos before, do you know if such tools exist, even if in other ecosystems?

[idan-at]

Sure.

Pants has it for Python
Yarn and PNPM for node
I think also cargo for rust, not sure

[pawamoy]

Interesting, thanks. I'll add https://blog.logrocket.com/managing-full-stack-monorepo-pnpm/ to the list, which explains how to use PNPM in a monorepo setup. PDM could get inspiration from this 🙂

Possibly related and interesting to you: #1505

[majidaldo]

I don't think so, but I'm sure there are [Python] tools or libraries that are based on Git and provide such functionality (showing/working on the set of changed files in a given repository).

dvc

[pawamoy]

You mean https://github.com/iterative/dvc 😕?

[DavidVujic]

Hi! 👋

I think all of the things listed is possible with PDM and additional tooling. I am the maintainer of the Python tools for the Polylith Architecture, and have just released support for repos that use PDM. Previously it was only Poetry, and later on Hatch.

In a Polylith workspace, you have source code and artifacts to build just as with any monorepo, but the developer experience is like if it were a single-project repo. All source code lives at the top-level, and the artifacts to build is basically a pyproject.toml for each project (that's what it is called in Polylith). Linting rules and things like MyPy configs are shared.

The tooling support includes visualization, diff based on changes and which projects are affected by changes. This is useful when determining what to deploy (such as in a CI environment) and when running tests.

I hope it is okay to join the discussion like this, and that you don't feel it's too much of marketing. I'm really excited about the new features supporting PDM and would very much like to hear your feedback. Here's the docs if you are interested and want to know more, or maybe we can continue talking here?