S3 Connection setup: Could not get access credentials

[aaj-synth]

Describe the bug

We are running clearml-helm-chart on our k8s cluster and trying to a submit a job which can access a s3 bucket for storing artifacts. We are providing apiserver pods access to our s3 bucket via apiserver service account, by adding the following annotation to it: eks.amazonaws.com/role-arn: <AWS-ROLE-WITH-BUCKET-ACCESS-POLICY>. On the apiserver pods, i can see the following under the containers.Environment section:

AWS_STS_REGIONAL_ENDPOINTS:                            regional
AWS_DEFAULT_REGION:                                    <REGION>
AWS_REGION:                                            <REGION>
AWS_ROLE_ARN:                                          <AWS-ROLE-WITH-BUCKET-ACCESS-POLICY>
AWS_WEB_IDENTITY_TOKEN_FILE:                           /var/run/secrets/eks.amazonaws.com/serviceaccount/token

As a test, i exec-ed in apiserver pod, and ran boto3 commands to access the S3 bucket in python shell:

import boto3; print("\n".join([obj['Key'] for obj in boto3.client('s3').list_objects_v2(Bucket='<AWS-BUCKET>').get('Contents', [])]))
import boto3; boto3.client('s3').upload_file('./some-file', '<AWS-BUCKET>', 'file')

The commands were able to access the S3 bucket and perform read and write operations. However, when i try to log artifacts in the same S3 bucket with the clearml api, i get the following error:

ValueError: Could not get access credentials for '<AWS-BUCKET>' , check configuration file ~/clearml.conf

My clearml.conf is straightforward and looks as follows:

sdk {
  aws {
    s3 {
      use_credentials_chain: true
    }
    boto3 {
      pool_connections: 20
      max_multipart_concurrency: 16
    }
  }
}

To reproduce

I'm simply trying to Task.init with output_uri set as the S3 bucket with the setup configured as described above.

Expected behaviour

What is the expected behaviour? What should've happened but didn't?

Environment

• Server type (self hosted \ app.clear.ml): Self Hosted Kubernetes
• ClearML SDK Version: Helm Chart Version -> 7.11.4
• ClearML Server Version (Only for self hosted). Can be found on the bottom right corner of the settings screen.
• Python Version: 3.11
• OS (Windows \ Linux \ Macos): macos

Related Discussion

#1017

[jkhenning]

Hi @aaj-synth,

When uploading artifacts the settings should be done on the client side, not the server side. The client (SDK in this case) communicated with S3 independently