Pod throwing permission denied when Volsync Src job runs

[scoutjohn]

Hi Team,
We have Apache nifi running on a custom Kubernetes environment.
In a GR setup, where nifi is running in 2 locations.
Nifi has PVC mounted into the container, we're trying to move the data from one location to another.

It is failing in this custom Kubernetes environment; we realized that the roles and SCC cannot be applied here as this is not and openshift platform. This environment has a PodSecurityPolicy instead of SecurityContextConstraint.

when the source job runs,

bash-4.4$ cd /opt/nifi/data/
bash-4.4$ ls -lRt
.:
total 44
drwxrwsr-x. 3 root 510 4096 Nov 5 12:53 conf_directory
drwxrwsr-x. 18 root 510 4096 Nov 5 12:52 state_directory
drwxrwsr-x. 3 root 510 4096 Nov 5 09:38 flowfile_directory
drwxrwsr-x. 1026 root 510 20480 Nov 5 09:38 content_directory
drwxrwsr-x. 2 root 510 4096 Nov 5 09:38 database_directory
drwxrwsr-x. 2 root 510 4096 Nov 5 09:38 user_conf_directory
drwxrwsr-x. 2 root 510 4096 Nov 5 09:37 provenance_directory
drwxrwsr-x. 3 root 40018 24 Nov 5 09:36 cache_directory

ls: cannot open directory './conf_directory': Permission denied
ls: cannot open directory './state_directory': Permission denied
ls: cannot open directory './flowfile_directory': Permission denied
ls: cannot open directory './content_directory': Permission denied
ls: cannot open directory './database_directory': Permission denied
ls: cannot open directory './user_conf_directory': Permission denied
ls: cannot open directory './provenance_directory': Permission denied

java.io.FileNotFoundException: /opt/nifi/data/state_directory/partition-0/1.journal (Permission denied) at java.base/java.io.FileOutputStream.open0(Native Method) at java.base/java.io.FileOutputStream.open(FileOutputStream.java:293) at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:235) at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:184) at org.wali.MinimalLockingWriteAheadLog$Partition.rollover(MinimalLockingWriteAheadLog.java:788) at org.wali.MinimalLockingWriteAheadLog.checkpoint(MinimalLockingWriteAheadLog.java:534)

the UI throws UncheckedIOException

java.io.UncheckedIOException: Set Verification Key State Failed at org.apache.nifi.web.security.jwt.key.service.StandardVerificationKeyService.setVerificationKey(StandardVerificationKeyService.java:142) at</p>

This error is only thrown when the source job runs, nifi container and UI becomes accessible in the window between the source job terminates and respawns. In that gap the pod gives below output

`bash-4.4$ ls -lFnR /opt/nifi/data/
/opt/nifi/data/:
total 44
drwxrwsr-x. 2 0 510 0 Sep 10 14:08 cache_directory/
drwxrwsr-x. 3 0 510 4096 Oct 10 19:29 conf_directory/
drwxrwsr-x. 1026 0 510 20480 Sep 10 14:12 content_directory/
drwxrwsr-x. 2 0 510 4096 Sep 10 14:12 database_directory/
drwxrwsr-x. 4 0 510 4096 Oct 10 19:29 flowfile_directory/
drwxrwsr-x. 2 0 510 4096 Sep 10 14:12 provenance_directory/
drwxrwsr-x. 18 0 510 4096 Oct 15 21:31 state_directory/
drwxrwsr-x. 2 0 510 4096 Sep 10 14:12 user_conf_directory/

/opt/nifi/data/cache_directory:
total 0

/opt/nifi/data/conf_directory:
total 12
drwxrwsr-x. 2 501 510 4096 Sep 10 14:12 extensions/
-rw-r--r--. 1 501 510 794 Oct 10 19:29 flow.json.gz
-rw-r--r--. 1 501 510 798 Oct 10 19:29 flow.xml.gz

/opt/nifi/data/conf_directory/extensions:
total 0

/opt/nifi/data/content_directory:
total 4096`

id

`bash-4.4$ id
uid=501 gid=510 groups=510

groups

bash-4.4$ cut -d: -f1,3 /etc/passwd
root:0
bin:1
daemon:2
adm:3
lp:4
sync:5
shutdown:6
halt:7
mail:8
operator:11
games:12
ftp:14
nobody:65534
dbus:81
systemd-coredump:999
systemd-resolve:193
sshd:74
nifi:1000

bash-4.4$ cut -d: -f1,3 /etc/group
root:0
bin:1
daemon:2
sys:3
adm:4
tty:5
disk:6
lp:7
mem:8
kmem:9
wheel:10
cdrom:11
mail:12
man:15
dialout:18
floppy:19
games:20
tape:33
video:39
ftp:50
lock:54
audio:63
users:100
nobody:65534
utmp:22
utempter:35
ssh_keys:999
dbus:81
input:998
kvm:36
render:997
systemd-journal:190
systemd-coredump:996
systemd-resolve:193
sshd:74
nifi:1000`

We have tested this in an openshift environment, and it works as expected on an it.
In the openshift environment the same command gives the expected output, and UI is accessible when the source job is running.

@JohnStrunk Can you please advise on how to proceed.

[tesshuflower]

It looks like the errors above are from your workload pod, not the VolSync pod? Are you saying when the VolSync source job runs, your workload loses access to the files?

Could you provide more info about the following:

• Is the workload in question running at the source or the destination?
• Info about your volsync replicationsource and replicationdestination, including what mover type are you using, are you using direct mode, your moverSecurityContext
• Are you running the source/dest in privileged mode?

[scoutjohn]

@tesshuflower thank you for responding,

• The issue is caused by the job running at the source,
• We're using Rsynctls in direct mode.
• Yes, we're running both source and destination in privileged mode

this is how the post job looks like

kubectl get job -nns-fm volsync-rsync-tls-src-ns-fm-acp-nifi-src-0 -oyaml
apiVersion: batch/v1
kind: Job
metadata:
  creationTimestamp: "2024-10-23T10:58:09Z"
  generation: 1
  labels:
app.kubernetes.io/created-by: volsync
    volsync.backube/cleanup: d453bdf9-99ab-4838-815f-cf8d66949e78
  name: volsync-rsync-tls-src-ns-fm-acp-nifi-src-0
  namespace: ns-fm
  ownerReferences:
  - apiVersion: volsync.backube/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicationSource
    name: ns-fm-acp-nifi-src-0
    uid: d453bdf9-99ab-4838-815f-cf8d66949e78
  resourceVersion: "205744341"
  uid: fe6f742b-37ba-4163-9941-dcbf8b628c96
spec:
  backoffLimit: 2
  completionMode: NonIndexed
  parallelism: 1
  selector:
    matchLabels:
      controller-uid: fe6f742b-37ba-4163-9941-dcbf8b628c96
  suspend: false
  template:
    metadata:
      creationTimestamp: null
      labels:
app.kubernetes.io/component: rsync-tls-mover
app.kubernetes.io/created-by: volsync
app.kubernetes.io/name: src-ns-fm-acp-nifi-src-0
app.kubernetes.io/part-of: volsync
        controller-uid: fe6f742b-37ba-4163-9941-dcbf8b628c96
        job-name: volsync-rsync-tls-src-ns-fm-acp-nifi-src-0
      name: volsync-rsync-tls-src-ns-fm-acp-nifi-src-0
    spec:
      containers:
      - command:
        - /bin/bash
        - -c
- /mover-rsync-tls/client.sh
        env:
        - name: DESTINATION_ADDRESS
value: 10.201.128.222
        - name: DESTINATION_PORT
          value: "44400"
        - name: PRIVILEGED_MOVER
          value: "1"
image: 10.201.0.133:30003/nac/volsync:0.8.0
        imagePullPolicy: IfNotPresent
        name: rsync-tls
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - DAC_OVERRIDE
            - CHOWN
            - FOWNER
            - SETGID
            drop:
            - ALL
          privileged: false
          readOnlyRootFilesystem: true
          runAsUser: 0
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /data
          name: data
        - mountPath: /keys
          name: keys
        - mountPath: /tmp
          name: tempdir
      dnsPolicy: ClusterFirst
      nodeSelector:
kubernetes.io/hostname: spteldc1-ncspd-workerbm-0
      restartPolicy: Never
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: volsync-src-ns-fm-acp-nifi-src-0
      serviceAccountName: volsync-src-ns-fm-acp-nifi-src-0
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoExecute
key: node.kubernetes.io/not-ready
        operator: Exists
        tolerationSeconds: 60
      - effect: NoExecute
key: node.kubernetes.io/unreachable
        operator: Exists
        tolerationSeconds: 60
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: ns-fm-acp-nifi-nifi-parent-pvc-ns-fm-acp-nifi-nifi-0
      - name: keys
        secret:
          defaultMode: 384
          secretName: ns-fm-acp-nifi-tls-key-secret
      - emptyDir:
          medium: Memory
        name: tempdir
status:
  active: 1
  startTime: "2024-10-23T10:58:09Z"

[tesshuflower]

Is there a reason you're using Direct mode? This means you will be copying files that are possibly being written as you copy them. If you want a crash-consistent copy, I would recommend you use Snapshot or Clone.

If you do need to use direct mode, have you tried not using privileged mode? Ideally you would use unprivileged and then set your moverSecurityContext to match what you use on your application workload.

I'm wondering if you are getting different seLinuxOptions when running the mover pod, which could cause selinux relabeling when the mover pod mounts the pvc.

It might be work looking on your user pod and the mover pod and seeing if they do not match i.e. the pod.spec.securityContext.seLinuxOptions.