Permission denied when using artifact as an input in non-root container

[jmbass]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

Consider the following Workflow template, which pulls an artifact:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: mywf
spec:
  entrypoint: clone
  templates:
    - name: clone
      inputs:
        artifacts:
          - name: repo
            path: /repo/
            git:
              repo: mygitlab.com/project/myrepogit
              branch: main
              usernameSecret:
                name: gitlab-creds
                key: user
              passwordSecret:
                name: gitlab-creds
                key: token
              singleBranch: true
      script:
        image: myimage:latest
        command: [python3.12, "-m"]
        args:
          - "args..."

Also, consider the following executor configuration:

executor:
  image:
    registry: filer-docker-registry.prod.crto.in
    repository: quay.io/argoproj/argoexec
    tag: "v3.5.11"
    pullPolicy: ""
  args: []
  env: []
  securityContext:
    readOnlyRootFilesystem: false
    runAsUser: 10002 # this user exists in the wf image, also tried with 8737
    runAsNonRoot: true
    allowPrivilegeEscalation: false
    capabilities:
      drop:
        - ALL

When running the workflow above, the following error occurs:

init time="2025-01-21T16:24:13.518Z" level=info msg="Starting Workflow Executor" version=v3.5.11
init time="2025-01-21T16:24:13.521Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
stream logs failed container "wait" in pod "mywf...." is waiting to start: PodInitializing for ns/mywf (wait)
init time="2025-01-21T16:24:13.521Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=siem podName=mywf... templateName=mywf version="&Version{Version:v3.5.11,BuildDate:2024-09-20T14:09:00Z,GitCommit:25bbb71cced32b671f9ad35f0ffd1f0ddb8226ee,GitTag:v3.5.11,GitTreeState:clean,GoVersion:go1.21.13,Compiler:gc,Platform:linux/amd64,}"
init time="2025-01-21T16:24:13.594Z" level=info msg="Loading script source to /argo/staging/script"
init time="2025-01-21T16:24:13.594Z" level=info msg="Start loading input artifacts..."
init time="2025-01-21T16:24:13.594Z" level=info msg="Downloading artifact: repo"
init time="2025-01-21T16:24:13.594Z" level=info msg="Load artifact" artifactName=repo duration="107.787µs" error="failed to clone \"gitlab/repo.git\": mkdir /mainctrfs/repo.tmp: permission denied" key=
init time="2025-01-21T16:24:13.594Z" level=error msg="executor error: artifact repo failed to load: failed to clone \"gitlab/repo.git\": mkdir /mainctrfs/repo.tmp: permission denied"
init time="2025-01-21T16:24:13.595Z" level=info msg="Alloc=10071 TotalAlloc=13614 Sys=25701 NumGC=3 Goroutines=4"
init time="2025-01-21T16:24:13.595Z" level=fatal msg="artifact repo failed to load: failed to clone \"gitlab/repo.git\": mkdir /mainctrfs/repo.tmp: permission denied"
stream logs failed container "main" in pod "mywdf..." is waiting to start: PodInitializing for ns/mywf.... (main)

What I've tried

This seems like a security context issue. I noted that the Argoexec mounts the mainctrfs itself, so it surprises me that it is unable to write to such directory. I tried mounting it again from the workflow, but of course, this results in the executor mounting /mainctrfs/mainctrfs/.

I also tried setting an fsGroup in the Executor security context to fix this, but this caused an error in the controller:

time="2025-01-22T10:11:15Z" level=info msg="index config" indexWorkflowSemaphoreKeys=true
time="2025-01-22T10:11:15Z" level=info msg="cron config" cronSyncPeriod=10s
time="2025-01-22T10:11:15Z" level=info msg="Memoization caches will be garbage-collected if they have not been hit after" gcAfterNotHitDuration=30s
time="2025-01-22T10:11:15.972Z" level=info msg="not enabling pprof debug endpoints"
time="2025-01-22T10:11:15.986Z" level=fatal msg="Failed to register watch for controller config map: error unmarshaling JSON: while decoding JSON: json: unknown field \"fsGroup\""
Stream closed EOF for siem/argo-workflows-workflow-controller-5768c7c869-r7v4r (controller)

(I believe that fsGroup should be supported nevertheless).

To me, running as root is not an option.

Looking for insights before forking and trying to fix!

Version(s)

v3.5.11

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

N/A

Logs from the workflow controller

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: mywf
spec:
  entrypoint: clone
  templates:
    - name: clone
      inputs:
        artifacts:
          - name: repo
            path: /repo/
            git:
              repo: mygitlab.com/project/myrepogit
              branch: main
              usernameSecret:
                name: gitlab-creds
                key: user
              passwordSecret:
                name: gitlab-creds
                key: token
              singleBranch: true
      script:
        image: myimage:latest
        command: [python3.12, "-m"]
        args:
          - "args..."
---
executor:
  image:
    registry: filer-docker-registry.prod.crto.in
    repository: quay.io/argoproj/argoexec
    tag: "v3.5.11"
    pullPolicy: ""
  args: []
  env: []
  securityContext:
    readOnlyRootFilesystem: false
    runAsUser: 10002 # this user exists in the wf image, also tried with 8737
    runAsNonRoot: true
    allowPrivilegeEscalation: false
    capabilities:
      drop:
        - ALL

Logs from in your workflow's wait container

init time="2025-01-21T16:24:13.518Z" level=info msg="Starting Workflow Executor" version=v3.5.11
init time="2025-01-21T16:24:13.521Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
stream logs failed container "wait" in pod "mywf...." is waiting to start: PodInitializing for ns/mywf (wait)
init time="2025-01-21T16:24:13.521Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=siem podName=mywf... templateName=mywf version="&Version{Version:v3.5.11,BuildDate:2024-09-20T14:09:00Z,GitCommit:25bbb71cced32b671f9ad35f0ffd1f0ddb8226ee,GitTag:v3.5.11,GitTreeState:clean,GoVersion:go1.21.13,Compiler:gc,Platform:linux/amd64,}"
init time="2025-01-21T16:24:13.594Z" level=info msg="Loading script source to /argo/staging/script"
init time="2025-01-21T16:24:13.594Z" level=info msg="Start loading input artifacts..."
init time="2025-01-21T16:24:13.594Z" level=info msg="Downloading artifact: repo"
init time="2025-01-21T16:24:13.594Z" level=info msg="Load artifact" artifactName=repo duration="107.787µs" error="failed to clone \"gitlab/repo.git\": mkdir /mainctrfs/repo.tmp: permission denied" key=
init time="2025-01-21T16:24:13.594Z" level=error msg="executor error: artifact repo failed to load: failed to clone \"gitlab/repo.git\": mkdir /mainctrfs/repo.tmp: permission denied"
init time="2025-01-21T16:24:13.595Z" level=info msg="Alloc=10071 TotalAlloc=13614 Sys=25701 NumGC=3 Goroutines=4"
init time="2025-01-21T16:24:13.595Z" level=fatal msg="artifact repo failed to load: failed to clone \"gitlab/repo.git\": mkdir /mainctrfs/repo.tmp: permission denied"
stream logs failed container "main" in pod "mywdf..." is waiting to start: PodInitializing for ns/mywf.... (main)