Strawpoll on storing rendered manifests in git

[danielloader]

Figured I'd ask the community and get a temperature check on what people have chosen to do, and ideally why.

I've got 20+ applications registered in Kargo and currently just abusing the update-argocd stage to change the targetRevision on the sources on the applications.

Note

All these are stored in a single repository rather than repository per project.

I'd like to switch to a rendered manifests solution, if only so the full GitOps state is recorded and available in the instance of the platform cluster running Kargo ever fails me and I need to rebuild it.

With that in mind I'm weighing up my options, there's pros and cons with each - and Github's WebUI constantly spamming the repository with the following, is quite annoying:

Ideally I'd be able to disable this on this repository, but Github hasn't budged on this feature request for years.

So the question really boils down to:

"How many branches do you use? and have you had any issues using fewer branches and sharing branches between multiple projects and or stages".

Some scenarios

Render everything into an output branch (say rendered)

Simple design, source your kustomization directories from the main branch, do your kustomize-set-image and kustomize-build steps and then push the resulting output into a ./{{stage}}/{{project}} directory.

Render into stage branches

Trade off between rendering to a branch and rendering to a {{stage}}/{{project}} branch.

Each stage has a branch, but the output is stored in a subdirectory on this branch per project.

Render into a branch per stage per project

More branches, but more isolation. Render into a fully isolated {{stage}}/{{project}} branch, and utilising git-clear stage to keep it "clean".

Give up on a monorepo for manifests and have a git repository per project, and use stages as branches

Basically the quickstart demo model, but feels very repository overhead intensive for 20-30 projects.

[RationalLemming]

We have a separate monorepo for all application manifests related to a product/team (which also maps to an individual Kargo project). Each team gets a separate monorepo. The manifests for each application/service is stored in separate sub-directories within the monorepo. We then have a separate branch per stage which contains the rendered manifests.

[danielloader]

Excellent to know, how many individual unrelated services or applications exist per project? One or more than one? Wondering how you are using warehouses and stages if the answer is more than one.

The web UI wasn't that friendly to multiple warehouses that were in a project but that wholly separate flows of promotions, through the same stages last time I tried.

[RationalLemming]

Our GitOps monorepo existed before Kargo. Therefore we've tried to fit Kargo into our existing workflow. The Kargo web UI has been a bit hard to align to our workflow so we have also changed our workflow slightly to try to fit with Kargo. This is definitely an area that we will revisit as Kargo matures because I'm still not entirely happy with what we've got. I tried multiple warehouses (and even multiple projects) but I couldn't get the behaviour that I was looking for with our monorepo. We've currently got a single Warehouse within each project with a subscription to the monorepo containing application/service manifests (and the Warehouse also has subscriptions to the image repository for each service). Thankfully the services are closely related so we promote all services together. This works for the current use case but we have other use cases (product teams) on other clusters that we haven't migrated to Kargo yet and our current Kargo design won't work for them.

[krancour]

@danielloader an option not on your list that solves the noise in your repo from constant pushes to dozens or hundreds of branches... use a separate repo for all that. The "source" for your configs lives in your main repo. The other repo is treated as storage for "compiled" (rendered) configs.

[danielloader]

Interesting additional option. The joys of a free form tool that's minimally opinionated.