K8s Pod CreateContainerError rootfs mounts caused: rootfs_linux.go:1180: mounting sysfs to rootfs

[xlw712]

I refer to the documentation now https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-k8s.md#kubernetes-distro-requirements install sysbox.
Installed via k8s, but found the following error when starting the pod
Environmental information
K8s=1.27.0
SysBox.image=sysbox-deploy-k8s:v0.6.2
K8s.Node.OS below

LSB Version:	core-11.1.0ubuntu4-noarch:security-11.1.0ubuntu4-noarch
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.3 LTS
Release:	22.04
Codename:	jammy

Pod Error Msg
Warning Failed 86s (x3 over 111s) kubelet (combined from similar events): Error: container create failed: time="2023-11-24T14:08:37+08:00" level=error msg="container_linux.go:424: starting container process caused: process_linux.go:607: container init caused: rootfs_linux.go:70: setting up rootfs mounts caused: rootfs_linux.go:1180: mounting "sysfs" to rootfs "/var/lib/sysbox/shiftfs/8286d1f5-12ac-4bec-b898-66181d388adb" at "/sys" caused: mount through procfd: operation not permitted"

sysbox-deploy pod enable

Is there any way? Very urgent Thanks

[ctalledo]

Hi @xlw712, thanks for giving Sysbox a try.

That error typically occurs when you miss the user-ns annotation in the pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: ubu-bio-systemd-docker
  annotations:
    io.kubernetes.cri-o.userns-mode: "auto:size=65536"         <<< HERE
spec:
  runtimeClassName: sysbox-runc
  containers:
  - name: ubu-bio-systemd-docker
    image: registry.nestybox.com/nestybox/ubuntu-bionic-systemd-docker
    command: ["/sbin/init"]
  restartPolicy: Never

Can you double check that annotation is present in your pod spec?

Thanks!

[xlw712]

Thanks you reply,I used the pod spec for annotations. here's my pod template.He's still reporting an error

apiVersion: v1
kind: Pod
metadata:
  name: pixiu-cicd-{{ POD_POSTFIX }}
  annotations:
    io.kubernetes.cri-o.userns-mode: "auto:size=65536"
  labels:
    app: web
spec:
  nodeSelector:
    using: pixiu_cicd
  runtimeClassName: sysbox-runc
  containers:
    - name: gocd-agent-container
      image: gocd-agent-docker-dind:aliunx3.8
      resources:
        requests:
          memory: 125000000Ki
          cpu: 28
        limits:
          memory: 125000000Ki
          cpu: 30

But before you could respond, I found a solution , It comes from #67

mount by changing solution, I'm Posting this in the hope that it will help others
Before change

findmnt | grep "sysfs"
├─/sys                                                                                                                       sysfs                                          sysfs       rw,nosuid,nodev,noexec,relatime

Fix Step

mount -o remount,rw /sys /sys

findmnt | grep "sysfs"
├─/sys                                                                                                                       sysfs                                          sysfs       rw,relatime

[xlw712]

Hi @ctalledo I started the pod using the method above,But I find that I still can't use docker related commands in contariner. Is there any documentation that can help me check?

[ctalledo]

Hi @xlw712,

mount -o remount,rw /sys /sys

That's interesting, not sure why /sys was mounted read-only in the sysbox pod, that should normally not be the case.

Regarding the latest error (i.e., dockerd not starting), looks like you need to start it manually (e.g., dockerd > /var/log/dockerd.log 2>&1&)?

I am not familiar with the gocd-agent-docker-dind:aliunx3.8, so don't know why dockerd is not starting automatically when you start the pod. Check the image's Dockerfile to see what the entrypoint is. For comparison, In the nestybox/ubuntu-focal-systemd-docker images we include systemd and that starts dockerd automatically.

[xlw712]

Yes. you right. I'am docker is not start.
After start is OK. Thanks

[ctalledo]

Ok cool; are you good now, or do you need further help?

[xlw712]

There are no new questions at the moment, if there are, I will open a new discussions。
Thanks again for your reply