Issues with authenticating to azure container registry

[akselleirv]

I'm trying to subscribe to a container changes in ACR and I have the following secret:

apiVersion: v1
kind: Secret
metadata:
  name: acr-pull
  namespace: kargo-my-project
  labels:
    kargo.akuity.io/secret-type: repo-creds
stringData:
  password: <repository-token>
  type: image
  url: <my-registry>.azurecr.io
  username: <token-name>

But it's failing with the following logs:

time="2023-10-13T11:08:46Z" level=debug msg="reconciling Stage" namespace=kargo-my-project stage=my-app-dev
time="2023-10-13T11:08:46Z" level=debug msg="found Stage" namespace=kargo-my-project stage=my-app-dev 
time="2023-10-13T11:08:46Z" level=debug msg="obtained credentials for image repo" namespace=kargo-my-project repo=my-registry.azurecr.io/my-app stage=my-app-dev
time="2023-10-13T11:08:47Z" level=error msg="error syncing Stage: error syncing image repo subscriptions: error getting latest suitable tag for image \"my-registry.azurecr.io/my-app\": error fetching tags for image \"my-registry.azurecr.io/my-app\": errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required, visit https://aka.ms/acr/authorization for more information.\n" namespace=kargo-my-project stage=my-app-dev
time="2023-10-13T11:08:47Z" level=debug msg="done reconciling Stage" namespace=kargo-my-project stage=my-app-dev

I have tested locally with curl that the token is valid:

curl -v -u "<token-name>:<repository-token>" "https://my-registry.azurecr.io/oauth2/token?service=my-registry.azurecr.io&scope=repository:my-app:metadata_read"

# Which returns: {"access_token":"<access_token>"}

curl -H "Authorization: <access_token>" https://my-registry.azurecr.io/v2/my-app/tags/list

# Which returns: {"name":"my-app","tags":["0.1.0"]}

Any ideas what could be wrong?

[krancour]

We re-use Argo CD Image Updater for this and they've got an open issue about this:

argoproj-labs/argocd-image-updater#550

If you look at their docs, they also don't list ACR as a registry they have validated Image Updater against:

https://argocd-image-updater.readthedocs.io/en/stable/configuration/registries/

So, unfortunately, I think the answer to this conundrum lies upstream.

[krancour]

Near term maybe we could add the CLI to the image. iirc, it depends on Python? I don't want our image to get too bloated, so we'll have to see how much weight it adds.

Longer term, we want to get off of Image Updater because of transitive dependencies it has that are holding us back from upgrading important modules like Kubernetes itself. That would be a good time to do something cleaner and more sustainable than using the az CLI as a library.

I'll keep an eye on that PR and see where we go from there once it's been merged.

[akselleirv]

I guess this issue would make it possible to add support for ACR in Kargo directly. I'll wait until that issue have made more progress.

[krancour]

Yep. I wanted to reference that issue, but was on my phone when I typed my previous response. One way or the other, we'll get there.

[BasJ93]

I see that #958 had it's last child issue merged. Has this resolved the connectivity issues with ACR?

And if so, how do I configure Kargo to look at an ACR repository, as following the example in Getting Started Guide results in ACR responding with 401.

[krancour]

@BasJ93 I'm not sure, tbh.

What version of Kargo are you running, btw? There have been major changes to our credential management in main, so if you happen to be running Kargo from main, you need to follow docs at main.kargo.akuity.io (instead of kargo.akuity.io).

Also, which of this dizzying array of auth options are you trying to use?

https://learn.microsoft.com/en-us/azure/container-registry/container-registry-authentication?tabs=azure-cli

I can try this on ACR myself, but it's been a while since I've done much with it, so it might be a bit before I get around to re-acquainting myself.