O_TMPFILE on Linux with non-supporting kernel ("/tmp: Operation not supported")

[glebsts]

Happens in some k8s environments for me, as host kernel seems to be used as Docker kernel:
I was struggling with this for last week and found similar issues in other projects, and just few days ago @kamalpreetSec reported it under clair issues

Error opening temporary file in /tmp: Operation not supported

Environment
Clair version/image: 4.7.3 / image: quay.io/projectquay/clair:4.7.3
Clair client name/version: clairctl 4.7.3
Host OS: Oracle Linux 7
Kernel (e.g. uname -a): 5.4.17-2136.304.4.1.el7uek.x86_64
Kubernetes version (use kubectl version): Docker Version: 19.03.11-ol
Network/Firewall setup: Corporate proxy

For me it happened with tmpfs and nfsfs in k8s cluster, both rootless and non-rootless container, alpine, debian and original ubi-minimal, default and 777 permissions.
Similar patches: https://github.com/libvips/libvips/pull/1155/files (original thread)

Pull request by @RTann is this repo, great job on introducing OS-specific temp file handling in #1140 . This time it is more corner-case'ish issue.

Proposed solution:
fallback to opening temporary dir/file without using unix.O_TMPFILE flag.

cc @hdonnay @kamalpreetSec

[hdonnay]

I'm disinclined to do anything about this without understanding how/why the option is disabled.

If the environment in the linked issue is as reported, there's no reason this should be happening.

[RTann]

The StackRox project has had success running ClairCore by setting the /tmp volume to emptyDir: {}. We found this resolved someone else's problem, too. Have you tried that?

[glebsts]

I find it's dangerous as cannot determine size of this space (empty dir can have limit, but not sure about usage), and in cluster it utilizes space of the node, which is bad, so it makes sense to provide a volume from special nfs server.

…

On Mon, Mar 18, 2024, 19:42 Ross Tannenbaum ***@***.***> wrote: The StackRox project <https://github.com/stackrox/stackrox/blob/4.4.0/image/templates/helm/shared/templates/02-scanner-v4-07-indexer-deployment.yaml#L145> has had success running ClairCore by setting the /tmp volume to emptyDir: {}. We found this resolved someone else's problem, too. Have you tried that? — Reply to this email directly, view it on GitHub <#1289 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACO3F7I5URNQJUNKVZHPXDYY4RRTAVCNFSM6AAAAABE3H424OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBUGU2TIOJRGQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[hdonnay]

NFS is not supported

[glebsts]

@hdonnay Could you please elaborate on that? I mean, I cannot find anything about it at Clair Documentation pages.
What's the problem with NFS (well, besides this O_TMPFILE issue)?

[glebsts]

I'm disinclined to do anything about this without understanding how/why the option is disabled.
If the environment in the linked issue is as reported, there's no reason this should be happening.

To clarify, in my case host OS is Oracle Linux.
Here are some evidences of O_TMPFILE support being a fequent issue:
https://www.google.com/search?q=linux+o_tmpfile+support
As Docker uses host's kernel, and not every Linux kernel supports O_TMPFILE flag, this eventually became a problem with running Clair on a wide variety of k8s nodes. But workaround is quite simple - fallback to opening temp file without using this flag, "in a classic way", so it is always easier to patch app than a host kernel.

As for emptyDir{} I'll give it a try, but disk space for docker is limited on nodes, and looking at Clair memory consumption issues, I don't feel confident about disk space consumption as well. Also just recently I happened to see a 17GB docker image (some machine learning involved), which made me think that 20GB that I put for temp db might be too few. Meaning I have to rely on virtually unlimited NFS volumes.

[glebsts]

And yes, quite expected, emptyDir does not make any difference because it is still accessed through host kernel so I get same error.

[Kieran-Muller]

We've just begun testing claircore v1.5.25 & clair v4.7.3 and we're also running into this issue:

• Kube v1.26
• Host os: Ubuntu 20.04
• kernel: 5.4.0-173-generic.
• ext4 filesystem

[hdonnay]

@hdonnay Could you please elaborate on that?

NFS doesn't support O_TMPFILE and is documented to have issues around file locking, making it a bad choice for scratch space like /tmp.

[hdonnay]

And yes, quite expected, emptyDir does not make any difference because it is still accessed through host kernel so I get same error.

There's no way to disable O_TMPFILE support, so either your vendor is doing some very invasive kernel patching and you should contact their support, or there's something odd about your environment. What does grep /tmp /proc/mounts return from inside the container?

[hdonnay]

We've just begun testing claircore v1.5.25 & clair v4.7.3 and we're also running into this issue:

What does grep /tmp /proc/mounts return from inside the container?

[cathyyoung]

What does grep /tmp /proc/mounts return from inside the container?

There is no output (catting /proc/mounts doesn't have any reference to /tmp). This is on a standard IBM Cloud k8s worker node on Ubuntu 20.04 running kubernetes 1.26.

[Kieran-Muller]

@cathyyoung FYI I rolled back the update, so that may be why it is not there.
I'll get an output from tomorrow and put it in here.

[glebsts]

NFS doesn't support O_TMPFILE and is documented to have issues around file locking, making it a bad choice for scratch space like /tmp.

I understand the nice functionality for scratch space provided by flag support, but there should be a fallback option. Cleaning up can be done with standard measures (aka cron), or by Clair explicitly cleaning after itself, and would be still a better option than non-working clair (currently returning HTTP 500).

[Kieran-Muller]

What does grep /tmp /proc/mounts return from inside the container?

Output does not contain a mention of /tmp

$ cat /proc/mounts
overlay / overlay ro,relatime,lowerdir=/var/data/cripersistentstorage/io.containerd.snapshotter.v1.overlayfs/snapshots/183/fs:/var/data/cripersistentstorage/io.containerd.snapshotter.v1.overlayfs/snapshots/155/fs:/var/data/cripersistentstorage/io.containerd.snapshotter.v1.overlayfs/snapshots/154/fs:/var/data/cripersistentstorage/io.containerd.snapshotter.v1.overlayfs/snapshots/153/fs:/var/data/cripersistentstorage/io.containerd.snapshotter.v1.overlayfs/snapshots/152/fs:/var/data/cripersistentstorage/io.containerd.snapshotter.v1.overlayfs/snapshots/147/fs:/var/data/cripersistentstorage/io.containerd.snapshotter.v1.overlayfs/snapshots/135/fs,upperdir=/var/data/cripersistentstorage/io.containerd.snapshotter.v1.overlayfs/snapshots/954/fs,workdir=/var/data/cripersistentstorage/io.containerd.snapshotter.v1.overlayfs/snapshots/954/work,xino=off 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev tmpfs rw,nosuid,size=65536k,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666 0 0
mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
sysfs /sys sysfs ro,nosuid,nodev,noexec,relatime 0 0
tmpfs /sys/fs/cgroup tmpfs rw,nosuid,nodev,noexec,relatime,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup ro,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup ro,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/blkio cgroup ro,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/perf_event cgroup ro,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup ro,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /sys/fs/cgroup/devices cgroup ro,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup ro,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/pids cgroup ro,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /sys/fs/cgroup/rdma cgroup ro,nosuid,nodev,noexec,relatime,rdma 0 0
cgroup /sys/fs/cgroup/memory cgroup ro,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/freezer cgroup ro,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/cpuset cgroup ro,nosuid,nodev,noexec,relatime,cpuset 0 0
shm /dev/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
/dev/mapper/docker_data /etc/resolv.conf ext4 ro,relatime 0 0
/dev/mapper/docker_data /etc/hostname ext4 ro,relatime 0 0
/dev/mapper/docker_data /dev/termination-log ext4 rw,relatime 0 0
/dev/mapper/docker_data /etc/hosts ext4 rw,relatime 0 0
tmpfs /etc/istio/pod tmpfs ro,relatime,size=22020096k 0 0
tmpfs /etc/istio/proxy tmpfs rw,relatime,size=22020096k 0 0
/dev/mapper/docker_data /run/secrets/istio ext4 ro,relatime 0 0
tmpfs /run/secrets/tokens tmpfs ro,relatime,size=22020096k 0 0
/dev/mapper/docker_data /var/lib/istio/data ext4 rw,relatime 0 0
/dev/mapper/docker_data /run/secrets/workload-spiffe-uds ext4 rw,relatime 0 0
/dev/mapper/docker_data /run/secrets/workload-spiffe-credentials ext4 rw,relatime 0 0
/dev/mapper/docker_data /run/secrets/credential-uds ext4 rw,relatime 0 0
tmpfs /run/secrets/kubernetes.io/serviceaccount tmpfs ro,relatime,size=22020096k 0 0
proc /proc/asound proc ro,nosuid,nodev,noexec,relatime 0 0
proc /proc/bus proc ro,nosuid,nodev,noexec,relatime 0 0
proc /proc/fs proc ro,nosuid,nodev,noexec,relatime 0 0
proc /proc/irq proc ro,nosuid,nodev,noexec,relatime 0 0
proc /proc/sys proc ro,nosuid,nodev,noexec,relatime 0 0
proc /proc/sysrq-trigger proc ro,nosuid,nodev,noexec,relatime 0 0
tmpfs /proc/acpi tmpfs ro,relatime 0 0
tmpfs /proc/kcore tmpfs rw,nosuid,size=65536k,mode=755 0 0
tmpfs /proc/keys tmpfs rw,nosuid,size=65536k,mode=755 0 0
tmpfs /proc/timer_list tmpfs rw,nosuid,size=65536k,mode=755 0 0
tmpfs /proc/sched_debug tmpfs rw,nosuid,size=65536k,mode=755 0 0
tmpfs /proc/scsi tmpfs ro,relatime 0 0
tmpfs /sys/firmware tmpfs ro,relatime 0 0

[hdonnay]

The linked PR adds a fallback path and additional documentation. This will be in the next release.

[cathyyoung]

That's great news @hdonnay, thanks for putting that fallback in as we're currently blocked on this. When do you anticipate the next release will be?