Error using Azure managed identity while storing artifacts

[shimittal123]

Instead of using storage account key, I am making connection using managed identity to store artifacts in azure storage account.
I have defined it under workflow-controller-configmap:

data:
  artifactRepository: |2
    archiveLogs: true
    azure:
      endpoint: <>
      container: blobcontainer
      blobNameFormat: "{{workflow.name}}/{{pod.name}}"
      useSDKCreds: true

I define managed identity and create federated credential which i associate with service account and use that service account in argo workflow:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    azure.workload.identity/client-id: $clientid
    azure.workload.identity/tenant-id: $tenantid
    aadpodidentitybinding: azc-storage-service-account-selector
  name: $saname
  namespace: $namespace

I create azure identity:

apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentity
metadata:
  name: azc-storage-identity
  namespace: <namespace>
spec:
  type: 0  # 0 for UserAssignedIdentity
  resourceID: /subscriptions/<subscription-id>/resourcegroups/"resourcegroupname"/Microsoft.ManagedIdentity/userAssignedIdentities/"User_assigned_identity"
  clientID: "client-id"

and azure identity binding:

apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentityBinding
metadata:
  name: azc-storage-identity-binding
  namespace: <namespace>
spec:
  azureIdentity: azc-storage-identity
  selector: azc-storage-service-account-selector

Workflow:

- name: whalesay
   outputs:
      artifacts:
         - name: message
            path: /tmp/hello_world.txt
      metadata:
        labels:
          azure.workload.identity/use: 'true'
      script:
        name: ''
        image: <azure_container_registry>/images/azc-cli:latest
        command:
          - bash
        env:
          - name: AZURE_LOG_LEVEL
            value: info
        resources:
          limits:
            cpu: 500m
            memory: 1Gi
          requests:
            cpu: 250m
            memory: 512Mi
        source: >
          set -eux
          az login --federated-token "$(cat $AZURE_FEDERATED_TOKEN_FILE)"
          --service-principal -u $AZURE_CLIENT_ID -t $AZURE_TENANT_ID
          echo "Login successful"
          echo "Checking /tmp directory"
          ls /tmp
          echo "Creating file"
          echo {{workflow.name}} | tee /tmp/hello_world.txt
          echo "Checking file creation"
          cat /tmp/hello_world.txt
          echo "Copying file to Azure"
          echo "File uploaded"
      retryStrategy:
        limit: '3'
        backoff:
          duration: '10'
          factor: '2'
      podSpecPatch: '{"serviceAccountName": "azc-storage-sa"}'

But in the end, all steps are failing in wait container by stating:

unable to upload file /tmp/argo/outputs/artifacts/message.tgz to Azure: ===== INTERNAL ERROR ===== DefaultAzureCredential: failed to acquire a token. Attempted credentials: EnvironmentCredential: incomplete environment variable configuration. Only AZURE_TENANT_ID and AZURE_CLIENT_ID are set ManagedIdentityCredential: IMDS token request timed out AzureCLICredential: fork/exec /bin/sh: no such file or directory

But when i am using:

az storage blob upload --account-name <storage_account>
          --container-name blobcontainer --file /tmp/hello_world.txt --name
          abcdef_hello_world.txt

under any workflow step, it is able to store artifacts without storage account key. But still at the end step gets failed by stating same message as wait container is failing for all workflows.

[tooptoop4]

the emissary image is distroless so it does not have shell let alone az cli on it. https://github.com/Azure/azure-sdk-for-go/blob/48dd0f41119aa74cbcaa91fa4bac178c1fd09a70/sdk/azidentity/default_azure_credential.go#L33-L74 shows what ways can be used to auth with azure. the error is being hit at https://github.com/Azure/azure-sdk-for-go/blob/aeb9fa45716ff84b6ba4f4013699a695022a6f22/sdk/azidentity/azure_cli_credential.go#L124-L141 perhaps u can try these env variables: https://github.com/Azure/azure-sdk-for-go/blob/09d9eaf461f9c94c428c66ed7808a72ceceee93d/sdk/azidentity/environment_credential.go#L39-L77