General Process for kargo.akuity.io/authorized-stage when an app contains multiple binaries

[avo-sepp]

Howdy all,

I have an ArgoCD application which has multiple container deployments. What's the correct way to apply the annotation which allows Kargo to interface with that application?

  annotations:
    kargo.akuity.io/authorized-stage: "myproject:backend-prod,myproject:frontend-prod"

I am going to try with a comma. But there isn't really great documentation on this. I'd like to see the way to do it correctly in the docs, if it's possible.

[avo-sepp]

Update: This did not work. I need a solution here.

[krancour]

There's a lot of history here. A comma delimited list isn't supported and never was. Wildcards, however, once were. They went pre-GA due both to technical challenges they presented and some edge cases where it proved to be a "foot gun." I think the "foot gun" problem has mostly been resolved through other measures over the last few releases, so there might be room to revisit this at some point, but before going to deep into that, let me ask...

If you have one Argo CD Application that's deploying multiple workloads, why model those as separate Stages in Kargo? On the surface, it seems like an odd choice. We more typically see the inverse of that -- one Stage coordinating multiple Argo CD Apps.

[avo-sepp]

How does a stage manage two different container freight at the same time? How does it know which one it is deploying to which fields in Git?

This does not seem like it is explained as a possible avenue in the documentation. You'll have to enlighten me.

[avo-sepp]

I think you're enlightening me to something that wasn't clear in the higher abstraction of the docs. But is becoming more clear as I read the CRD docs themselves.

[avo-sepp]

kind: Stage
metadata:
  name: dev
  namespace: doc-suite
spec:
  requestedFreight:
  - origin:
      kind: Warehouse
      name: tool-backend
    sources:
      direct: true
  - origin:
      kind: Warehouse
      name: tool-frontend
    sources:
      direct: true
  promotionTemplate:
    spec:
      steps:
      - uses: git-clone
        config:
          repoURL: <git repo>
          checkout:
          - branch: main
            path: ./src
      - uses: kustomize-set-image
        as: update-backend-image
        config:
          path: ./src/manifests/dev
          images:
          - image: <backend registry base url>/<uri>
            useDigest: true
      - uses: kustomize-set-image
        as: update-frontend-image
        config:
          path: ./src/manifests/dev
          images:
          - image: <frontend-image url>
            useDigest: true
      - uses: git-commit
        as: commit
        config:
          path: ./src/manifests/dev
          messageFromSteps:
          - update-backend-image
          - update-frontend-image
      - uses: git-push
        config:
          path: ./src/manifests/dev
      - uses: argocd-update
        config:
          apps:
          - name: <app-name>
            sources:
            - repoURL: <repo>

Are you saying this is valid? It would know which freight to apply in the commit stages?

[avo-sepp]

Apparently not valid.

[avo-sepp]

I got it working. This definitely needs better docs. This is not at all what I expected was possible.

I also found some bugs. As seen by the runtime error. It turns out the first time I try to add freight, it failed. It didn't know what to do with the other freight. I had to go back and add the second freight, which was successful, for the whole thing to become successful going forward.

Definitely a setup I didn't expect, but ended up being possible.

[krancour]

Whoa. Hold on. I appreciate and admire your enthusiasm for troubleshooting this yourself, but I'm not sure you ended up where you really meant to. I want to make sure you end up with what will actually serve your use case best, so please help me understand it...

As far as I can tell, what you want is extremely straightforward. You have one (little a) application made up of two components -- a front end and a back end. Common. There's an image for each of these...

Now I don't know if you're new to Argo CD or if you're just getting started with it, but either way, I think you've done the correct and conventional thing for that scenario. You have one Argo CD (big A) Application resource that manages the deployment of both components. This is revealing, as it seems to indicate that your intention is to always deploy both components as a unit. Again, pretty normal...

Inexplicably, however, you've landed in a place where you have two different Warehouses where one subscribes to the front end repo and the other to the back end repo. This is where I think you've deviated from your intentions...

A Warehouse produces Freight. Freight is a meta-artifact -- a collection of references to other artifacts. For all intents and purposes, you may think of it as a box of artifacts. What you're doing right now is putting each image in its own box. I don't think you meant to do that. If you're deploying both components with one App, I think you want rather progress the pair of images through the pipeline as a unit... so they should be together in one box.

This is the conventional pattern. Things that should stick together should be in the same box.

Unless I've misunderstood what you're doing, use just one Warehouse that subscribes to images in both repos. This will produce Freight containing a pair of images and they will move through the pipeline together. Your Stages only need to request Freight from that one Warehouse. The rest of your dev Stage (i.e. the promotion steps you have defined) looks pretty good.

I think you're only a few lines off from what you really wanted to do, but you accidentally stumbled into a much more advanced scenario where you are progressing two different "boxes" through the same Stages independently of one another. I don't think that's what you wanted.

fwiw, the runtime error appears to be something we failed to catch, but the root cause is because the promotion process you defined fails when it's missing one of the two "boxes" it needs. Again... you somehow stumbled into something pretty advanced.

As far as docs go, we know they need substantial improvement and that is an incremental and ongoing process.

Please let me know if I've understood your use case correctly, if the explanation makes sense, and if this guidance resolves things. 🙏

[avo-sepp]

Hmm... no I intend to have both images be updated separately. I don't think that's uncommon at all. Often these need to be deployed together because they may need to trigger actions based on some shared element. Like a configmap or secret with connection strings or keys, testing, or automation. So they need to be in one Argo app. But they have different release cycles from the same dev team.

So, I'd like to be able to manage n-number of images separately. In that I don't always update the front end and backend as a pair. To me that sounds like running multiple freight lines through a single ArgoCD app. Which I've proved Kargo can do, even if it complains a little.

I had this working as two entirely separate DAGs in Kargo. But then I ran into the ArgoCD-update security error which started this thread.

However, the way I wrote the new Kargo stages as commented above, works as intended. Provided some initial errors coming out of Kargo.

[krancour]

no I intend to have both images be updated separately.

Thank you for clarifying.

I don't think that's uncommon at all.

I agree it's not.

Often these need to be deployed together

Since they're deployed by just one Argo CD App they are in fact always deployed together.

So your requirement is to be able to update one or the other (or sometimes both?) of the pair of images that will always be deployed together.

To me that sounds like running multiple freight lines through a single ArgoCD app.

Maybe. Before concluding that, let's look at the one Warehouse scenario in more detail using a reference pipeline that looks like dev --> qa --> prod.

1. Start with front end v1 and back end v1 in all stages. We'll denote the Freight referencing this combination of images as F1B1.

2. CI spits out an F2.

3. The single Warehouse finds the latest images for each component and creates Freight F2B1, in which only one in the pair of images has changed WRT what's in dev. It's (auto?)promoted into dev. If we like what we see there, we might promote F2B1 to qa.

4. After some time, CI spits out a B2.

5. The single Warehouse now creates Freight F2B2. Again, only one in the pair of images has changed WRT what's in dev. It's (auto?)promoted into dev. Let's assume we like what we see there and are considering promoting F2B2 into qa.

This is where we need to stop and ask ourselves what we care about. Freight F2B2 is the shiniest new pair of images that we hope might make it all the way to prod. In relative terms, are we now less enthused about F2B1? F2B1 may remain "ahead" of F2B2 in the pipeline, perhaps even making it all the way to prod hours or days before F2B2, but with any problems discovered with F2B2 along the way notwithstanding, whether it should happens in prod or sooner than that, when F2B2 overtakes F2B1 in the pipeline, do we see it as having fully superseded F2B1? The answer to all of these is often yes. This is a very environment-centric PoV wherein our focus is on promoting a combination of artifacts that's working well in one Stage on to the next one.

Note Freight F2B1 still exists, of course. If anything about F2B2 should prove problematic, any Stage that once hosted F2B1 can be restored to that.

The only thing we don't have at this point is Freight with the combination of F1 + B2. That's just not the order that things happened in. If we should somehow decide that combination of artifacts is of interest to us, it's easy to manually create F1B2 through the UI.

This might be pretty much what you wanted.

So why ever have multiple pipelines flowing through some or all Stages? It is something that typically only comes into play for much more complex scenarios. Perhaps not all artifacts need to take the same path from left to right. Perhaps some Stages require artifacts that others do not. Perhaps you frequently encounter the previously mentioned scenario where the shiniest new Freight is F2B2 and for some reason you want to promote a non existent F1B2 into some Stage. Being a frequent occurrence, you don't want to have to manually create F1B2 each time such a need pops up, so you decide to split things out so Fs and Bs flow from left to right on their own separate tracks. Even then, those can often just be two entirely disconnected DAGs (although not in your case, as we've established). I wouldn't say we've observed any of these scenarios to be especially common.

I'm glad you started this convo, because it's clear that in the multi-pipeline scenario it might be the case that we're poorly handling the scenario where a user-defined promotion process is intolerant of missing n of m pieces of Freight that it requires. But my main point delving into all of this is to help you avoid complexities that you may not need. I might suggest trying this with a single Warehouse to discover what, if any, pain points drive you toward a more complex option.

[avo-sepp]

I think one of my takeaways is that I'm still new to Kargo, and there are some capabilities that are not yet documented. Like, single warehouse with multiple freight options. So I’m going to give that a shot regardless. I'll learn something new. Form new plans.

I do like the approach of having two DAGs in this situation. I think for the end user, it looks more appealing and clear. This is actually what we did prior to the decision to add the argocd-update step, which needed the permission added. If the permission was able to select multiple Kargo project's and stages in a single app, either by glob or comma, then I could solve this by adding a separate DAG for each container flow.

I also think handling the usecase I tried here is interesting too. But, if the stage is missing either F or B, it shouldn't error out. Or maybe it should give a warning. That one of the freight slots remained empty after the operation, but the other one succeeded.

I'm going to re-read your last reply a few times. We're dealing with some production issues today that have distracted me from committing my brain fully to this. But it sounds like we're on the same page.

[krancour]

I do like the approach of having two DAGs in this situation.

Yes. I can see why that appeals. Users (and even maintainers) are constantly discovering new and useful patterns.

Documenting patterns that address various common use cases is one of my own most important tasks as soon as we wrap up the current release.

I also remain amenable to revisiting how necessary the current restrictions on the kargo.akuity.io/authorized-stage are at this point in time. A secondary use of that annotation is that when we observe a change in an App's status, that Annotation tells us what Stage needs to be re-reconciled to account for the latest changes in App health or sync state. You can imagine the problem that * presented. We'd have to requeue every Stage and most of it would be busy work. But a finite comma-delimited list of Stages probably wouldn't be so bad.

But, if the stage is missing either F or B, it shouldn't error out. Or maybe it should give a warning.

Yes. I want to improve this. I don't want to stop things in their tracks when this happens because some promotion processes may actually be more tolerant of some artifacts being missing. I think an error if the promotion process can't tolerate a missing artifact is the right thing -- but it shouldn't be a nil pointer dereference. I need to figure out what's slipping through the cracks and make sure the error message is more clear when this has happened.

fwiw, this is a rare occurrence. Each of your Stages is going to be missing some stuff only on the very first Promotion and then that problem won't happen again. I'm sure that's part of why we haven't already looked at this more closely.

Anyway... I think there's actionable stuff here and I will open issues for each as soon as I've got a moment.

So I’m going to give that a shot regardless.

lmk how it works out.

[avo-sepp]

I did it the way you told me. I switched the Kargo project to use a single Freight that is built with multiple binaries. I am now enlightened. A whole aspect of Kargo suddenly made sense to me.

Please put this in your docs. Please. I beg you. You have a great app it just lacks a user guide. Even if it's just a bunch of named examples. 🙏🏻

I still would like to have the ability to move multiple freight through the same stage. Either with separate DAGs or the same DAG. There are cases where this might be the working model. The more open ended the tool is the more uses I can find for it.

Thank you so much for being so helpful in this thread.

[krancour]

You have a great app it just lacks a user guide. Even if it's just a bunch of named examples.

We're actively working on this right now.