[Question] AKS cannot pull image from ACR with kubelet identity - 401 Unauthorized

[hoangquy4897]

Describe scenario
After using service principal for AKS for a year, I switched to user-assigned managed identity. Everything is working well except pulling image from ACR, here is the error:

Failed to pull image "myacr.azurecr.io/fluent/fluent-bit:3.1.6":[rpc error: code = Unknown desc = failed to pull and unpack image "myacr.azurecr.io/fluent/fluent-bit:3.1.6": failed to resove reference "myacr.azurecr.io/fluent/fluent-bit:3.1.6": failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://myacr.azurecr.io/oauth2/token?scope=repository%3Afluent%2Ffluent-bit%3Apull&service=myacr.azurecr.io: 401 Unauthorized, rpc error: code = Unknown desc = failed to pull and unpack image "myacr.azurecr.io/fluent/fluent-bit:3.1.6": failed to resolve reference "myacr.azurecr.io/fluent/fluent-bit:3.1.6": failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://myacr.azurecr.io/oauth2/token?scope=repository%3Afluent%2Ffluent-bit%3Apull&service=myacr.azurecr.io: 401 Unauthorized]

My resources meet all the requirements:

• AKS already have kubelet identity assigned.
• Kubelet identity was assigned AcrPull role with scope of exactly ACR.
• Kubelet identity was associated with VMSS of AKS node pools.
• There is no connection issue between ACR and AKS (It worked if I use secret in AKS)
• Service principal client id on AKS is msi.

I tried to follow these documents:

• https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/extensions/cannot-pull-image-from-acr-to-aks-cluster
• https://learn.microsoft.com/en-us/azure/aks/use-managed-identity

However, it didn't work. I even tried to scale new node on node pool.

Question
It seems like this is a bug, anyone have solution for this?

[PixelRobots]

I would open a support request for this issue. You will probably get a faster response.

[hoangquy4897]

Anything idea on this? I can provide the result of commands to check if my configuration matched the MS doc.