cli login with kubeconfig lacks permissions compared to sso

[anttijman]

Checklist

• I've searched the issue queue to verify this is not a duplicate bug report.
• I've included steps to reproduce the bug.
• I've pasted the output of kargo version.
• I've pasted logs, if applicable.

Description

I have granted our user all applicable rules to a project "kargo-podinfo". When this user logs in with the cli:
kargo login <url> --kubeconfig and then tries kargo cli cmds for example:
kargo get stage .. outputs an error: Error: permission_denied: namespaces "kargo-podinfo" is forbidden: get is not permitted
or kargo promote .. outputs an error: Error: promote stage subscribers: permission_denied: namespaces "kargo-podinfo" is forbidden: get is not permitted

On the other hand, when this same user logs in with --sso, these same commands work. However, this same user is in oidc viewers.
Also:

• kargo cli cmds work with --kubeconfig login for me, who has kube cluster admins.
• kargo-podinfo kargo project is created to a different namespace than rest of the kargo resources for the kargo-podinfo project.

What am I missing?

Screenshots

Steps to Reproduce

• Have AKS with kargo setup and Microsoft Entra ID authentication with Kubernetes RBAC
• Have MS Entra ID user with:
  • Azure Kubernetes Service Cluster User Role Azure RBAC
  • All rules to a kargo project
• Get az aks creds and then convert the config with kubelogin convert-kubeconfig -l azurecli
• Use kargo cli to login the user with --kubeconfig: kargo login <url> --kubeconfig
• Try to do something with kargo cli, for example: kargo get stage dev --project=kargo-podinfo

Version

Client Version: v1.2.0

Logs

time="2025-01-20T11:31:58Z" level=error msg="finished unary call" connect.code=permission_denied connect.duration=6.03373ms connect.method=GetStage connect.service=akuity.io.kargo.service.v1alpha1.KargoService connect.start_time="2025-01-20T11:31:58Z" error="permission_denied: namespaces \"kargo-podinfo\" is forbidden: get is not permitted"

[krancour]

When you "log in" with your kubeconfig, you are passing the token belonging to a cluster user with any request to the Kargo API server. It will not recognize the token and take a guess that the Kubernetes API server might. It will use that token for any communication with the Kubernetes API server in the course of a given request to the Kargo API server.

So... what that user can or can't do is entirely a matter of how you've configured RBAC with the cluster. No Kargo configuration comes into play at all.

[anttijman]

When you "log in" with your kubeconfig, you are passing the token belonging to a cluster user with any request to the Kargo API server. It will not recognize the token and take a guess that the Kubernetes API server might. It will use that token for any communication with the Kubernetes API server in the course of a given request to the Kargo API server.

So... what that user can or can't do is entirely a matter of how you've configured RBAC with the cluster. No Kargo configuration comes into play at all.

Thank you for the quick reply!

I guess my issue is then to understand what kind of kube RBAC kargo get stage .. & kargo promote .. cmds require.
Both of these cmds error with: permission_denied: namespaces "kargo-podinfo" is forbidden: get is not permitted even though I have granted get on namespaces at the cluster level on top of permissions to kargo resources.

[krancour]

If you believe you have granted permission to get on namespaces and you are getting that error, then you have misconfigured RBAC.

Look at the predefined kargo-viewer and kargo-admin ClusterRoles to understand... or just bind your user to one of those roles.

[anttijman]

If you believe you have granted permission to get on namespaces and you are getting that error, then you have misconfigured RBAC.

Look at the predefined kargo-viewer and kargo-admin ClusterRoles to understand... or just bind your user to one of those roles.

Yes it is possible that I have misconfigured my RBAC.
According to your suggestion, I applied a ClusterRoleBinding to kargo-viewer ClusterRole:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kargo-viewer-test
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kargo-viewer
subjects:
  - kind: User
    name: PLACEHOLDER_AZURE_MANAGED_IDENTITY_PRINCIPAL_ID
    apiGroup: rbac.authorization.k8s.io

And still getting Error: permission_denied: namespaces "kargo-podinfo" is forbidden: get is not permitted.
on kargo get stage ..

I have granted other permissions as well and I am testing what I can do with kubectl auth can-i and the MI has quite extensive permissions.. On top of the kargo-viewer ClusterRoleBinding, I have also done RoleBinding for the user in the kargo-podinfo namespace to the ClusterRole view.

For example, checking what permissions are granted on the whole cluster:

kubectl auth can-i --list

And furthermore, I can get the stage's freight just fine with kubectl:
kubectl get stage dev -n kargo-podinfo -o=jsonpath='{.status.freightSummary}'
but kargo get stage fails with the before mentioned namespaces error:
kargo get stage dev --project=kargo-podinfo -o=jsonpath='{.status.freightSummary}'

I am considering trying to do the promotion also with kubectl instead of kargo cli..

[krancour]

Ok... you've certainly done your due diligence in troubleshooting here, which I very much appreciate. I will look into this to see what I can find.

[krancour]

Oh... your kubeconfig...

Does it use a token for authentication or a client certificate?

This feature does not work at all with certificates, which means it also doesn't work with common local clusters like kind or k3d.

[anttijman]

Oh... your kubeconfig...

Does it use a token for authentication or a client certificate?

This feature does not work at all with certificates, which means it also doesn't work with common local clusters like kind or k3d.

I am pretty sure It is using a token. I am using AKS and I do (as MI)
az aks get-credentials ...
kubelogin convert-kubeconfig -l azurecli
kargo login .. etc.

Also this --kubeconfig kargo login works for me and I am a cluster admin and I do the same thing for kubeconfig.

edit I mean kargo cmds works (kargo get stage, kargo promote) for me when i kargo login with --kubeconfig.

[krancour]

Also this --kubeconfig kargo login works for me and I am a cluster admin

I mean kargo cmds works

I see this as evidence that it's working.

subjects:
  - kind: User
    name: PLACEHOLDER_AZURE_MANAGED_IDENTITY_PRINCIPAL_ID
    apiGroup: rbac.authorization.k8s.io

I'm going to go out on a limb... is it possible that the name you're including here looks like a human-friendly username or email address? Unless I am much mistaken, with Entra you need to put an object ID here. (At least that's the way it worked back when it was still called AAD.)

[anttijman]

``> > Also this --kubeconfig kargo login works for me and I am a cluster admin

I mean kargo cmds works

I see this as evidence that it's working.

subjects:
  - kind: User
    name: PLACEHOLDER_AZURE_MANAGED_IDENTITY_PRINCIPAL_ID
    apiGroup: rbac.authorization.k8s.io

I'm going to go out on a limb... is it possible that the name you're including here looks like a human-friendly username or email address? Unless I am much mistaken, with Entra you need to put an object ID here. (At least that's the way it worked back when it was still called AAD.)

It is the Object ID of the MI:

We are using other MIs in the same way as I am trying to do here with kargo. We grant the MI Azure Kubernetes Service Cluster User Role to the AKS so it can gets creds, and then we use kube RBAC to grant permissions to the cluster resources.

But I was finally able to test with a human user. I added a human user (email) to the subjects of the ClusterRoleBinding with kargo-viewer ClusterRole.
Then he did the same steps as me:

az aks get-credentials ... 
kubelogin convert-kubeconfig -l azurecli
kargo login .. --kubeconfig
kargo get stage .. 

And it succeeded.

So the problem seems to be with the MI after all. However, I am still puzzled about it because the same solution works for our other use cases with MIs..
Also the kubectl auth can-i tests prove that the RBAC is there for the MI, and kubectl get stage .. works.

[krancour]

Keep in mind that auth can-i is only testing RBAC rules. This may point to the real difficulty being authenticating as that principal.

Can you show me what the MI's kubeconfig looks like, with sensitive data redacted, of course?

[anttijman]

Keep in mind that auth can-i is only testing RBAC rules. This may point to the real difficulty being authenticating as that principal.

Can you show me what the MI's kubeconfig looks like, with sensitive data redacted, of course?

Here's the kubeconfig from just before trying the kubectl & kargo cmds. Sorry for the format, im on mobile.

===========KUBECTL CONFIG VIEW=============
apiVersion: v1
clusters:
- cluster:
   certificate-authority-data: DATA+OMITTED
   server: https://name-of-the-cluster-…
 name: name-of-the-cluster
contexts:
- context:
   cluster: name-of-the-cluster
   user: clusterUser_…
 name: name-of-the-cluster
current-context: name-of-the-cluster
kind: Config
preferences: {}
users:
- name: clusterUser_…
 user:
   exec:
     apiVersion: client.authentication.k8s.io/v1beta1
     args:
     - get-token
     - --login
     - azurecli
     - --server-id
     - 6dae42f8-43….
     command: kubelogin
     env: null
     installHint: |2

       kubelogin is not installed which is required to connect to AAD enabled cluster.

       To learn more, please go to https://aka.ms/aks/kubelogin
     interactiveMode: IfAvailable
     provideClusterInfo: false

It looks the same for me as well.

I might still test out using workload-identity for the kubelogin but i can not use it with our current federation of choice. So it would require some effort. And I havent tested it yet because current setup works for all of our other use cases.

[krancour]

installHint: |2

       kubelogin is not installed which is required to connect to AAD enabled cluster.

       To learn more, please go to https://aka.ms/aks/kubelogin

[anttijman]

installHint: |2

       kubelogin is not installed which is required to connect to AAD enabled cluster.

       To learn more, please go to https://aka.ms/aks/kubelogin

kubelogin is installed though. I'm not sure why it says that.

I did a promotion by creating a promotion resource with kubectl in the workflow with same RBAC. It seemingly worked fine until at some point, it didnt verify the stage anymore after a promotion. The promotion itself still worked but the UI never refreshed and it didnt verify the stage after promotion. I resolved the broken state by deleting the stage resource.

Seemed like it ended up in the broken state after another user tried to promote from UI or by creating another promotion resource with kubectl. Back to back promotions by the workflow seemed to work. Although, I didnt do extensive testing.

I might just drop this promotion from workflow idea. My goal is to promote automatically downstream after test automation has successfully passed.

[krancour]

I did a promotion by creating a promotion resource with kubectl in the workflow with same RBAC. It seemingly worked fine until at some point, it didnt verify the stage anymore after a promotion. The promotion itself still worked but the UI never refreshed and it didnt verify the stage after promotion. I resolved the broken state by deleting the stage resource.

Seemed like it ended up in the broken state after another user tried to promote from UI or by creating another promotion resource with kubectl. Back to back promotions by the workflow seemed to work. Although, I didnt do extensive testing.

Just to try and keep the issue focused, none of this seems related to the permissions issue.

kubelogin is installed though. I'm not sure why it says that.

For reference, here is how we get the token, very indirectly, from your kubeconfig. This convoluted process is only used specifically to be compatible with cases where the token is obtained via a plugin.

https://github.com/akuity/kargo/blob/main/internal/kubeclient/auth.go

I believe you that it is installed, but the kube client for sure does not think it is.

PATH issue?

[anttijman]

Hey, finally made some progress thanks to your link. Before it actually gets the creds, it gets the config and from GetConfig, I noticed the config precedence.

So what I did is, I paused the workflow run with sleep, connected to the runner pod (we are using github self-hosted arc runners) retrieved the bearer token with kargo config view --raw and then inspected that token at https://jwt.ms/ and found out that token belonged to the runner and not the MI.
That was according to the config precedence (In-cluster config if running in cluster over $HOME/.kube/config), then in the workflow, I set KUBECONFIG env to the default kubeconfig location and after that kargo login used the MI's kubeconfig. Then the kargo cmds started to work.

I dont know if there would be any ways to improve this. Maybe if we could give a path to the kubeconfig file in kargo login cmd. E.g. kargo login URL --kubeconfig=path/to/kubeconf.

Anyway thanks for helping, much appreciated. Should I close this issue?

P.S. I still had to do ClusterRoleBinding to namespaces (get) for the kargo promote to work. Not so problematic but I have to do two role bindings

[krancour]

First and foremost, I'm happy to hear this helped and that you're back in business.

kargo login URL --kubeconfig=path/to/kubeconf

Does KUBECONFIG=path/to/kubeconf login URL --kubeconfig not work?

Last, and only as an fyi (not finger-wagging), using the kargo CLI from a CI pipeline isn't something we encourage. Explanation here:

https://docs.kargo.io/new-docs/faqs#how-do-i-integrate-kargo-into-my-ci-pipelines

[anttijman]

First and foremost, I'm happy to hear this helped and that you're back in business.

kargo login URL --kubeconfig=path/to/kubeconf

Does KUBECONFIG=path/to/kubeconf login URL --kubeconfig not work?

Last, and only as an fyi (not finger-wagging), using the kargo CLI from a CI pipeline isn't something we encourage. Explanation here:

https://docs.kargo.io/new-docs/faqs#how-do-i-integrate-kargo-into-my-ci-pipelines

Yeah I've acknowledged that this isnt the best way to go about promotions. To my understanding, the way to go would be to use stage verifications and implement AnalysisTemplate that would take care of verifying that, for example, test automation has passed for that stage, then together with custom verification we could use autopromotion. However we are not using argo resources at all. Not saying that it couldnt still be done.
We are also waiting for new features in future releases that would help with autopromotion.

I will test this: KUBECONFIG=path/to/kubeconf login URL --kubeconfig. Thanks for the tip.