ci: add test for rootful docker

[vsoch]

I am finding with testing that the networking between hosts does not work when we are running in rootful. I was testing this because using nvidia devices does work with rootful, but once I got to the stop of needing pods to communicate, there was no communication.

I am not sure about the error, but this test should reproduce it in CI. Note that to enable this we use the docker-rootful template provided by lima (@AkihiroSuda you have thought of all things)! The main changes here are to add this test to the matrix, and ensure that in the different install scripts, we largely do nothing if the container runtime is docker-rootful.

Related to #365 but does not fix it, only demonstrates it.

[vsoch]

Note that I've seen two variants of this error - either an operation timeout (the result here):

Or that the address is not reachable / bad (what I've seen in production and my researchapps testing CI):

[AkihiroSuda]

I'd prefer this form

- lima_template: template://ubuntu-24.04
  container_engine: docker
  rootfull: 1

[AkihiroSuda]

I prefer the variables to be immutable through the entire lifecycle of the test.
So, another variable like ROOTFUL=1 should be introduced.

[AkihiroSuda]

Thanks, I confirmed that this issue happens on my local machines too, but I haven't identified the cause.

Tested with Docker v28 and v27.5.1, on Ubuntu 24.04.1 (ARM64).

I think it was working in the past?

[AkihiroSuda]

ICMP and DNS still seems to work, but TCP across the nodes seems broken?

VXLAN packets are apparently sent and received on each of the VMs, though. (Run tcpdump udp).

Apparently, the receiver VM seems refusing to route the VXLAN packets to the usernetes-node-1 container where kubelet, flannel, etc. are running in.

[AkihiroSuda]

Found a workaround: execute ethtool --offload eth0 tx-checksum-ip-generic off in usernetes-node-1 container

[thaJeztah]

Any eyes needed here from the Moby networking folks? (I know they're pretty busy currently, but if it's useful I can try ask them if they have time to spare to give it eyes)

[vsoch]

@AkihiroSuda do you remember the last time you tested with it working? In recent memory we had updates to flannel, the underlying kind node (Kubernetes version), and (for me) at some point last year the additional make sync-external-ip was added. If we can reproduce a previously working version it could be a good strategy to debug (to compare to).

[vsoch]

oh wow, this is really interesting!

Not sure if this is expected, but this looks to be a warning in the failed nerdctl setup:

Warning: 7m[WARNING] buildkitd has access to images in "buildkit" namespace by default. If you want to give buildkitd access to the images in "default" namespace, run this command with CONTAINERD_NAMESPACE=default

[AkihiroSuda]

The ethtool --offload eth0 tx-checksum-ip-generic off rule can be probably appended here:

usernetes/Dockerfile.d/etc_udev_rules.d_90-flannel.rules

Lines 1 to 5 in b259da8

	# Correct UDP checksums for VXLAN behind NAT
	# https://github.com/flannel-io/flannel/issues/1279
	# https://github.com/kubernetes/kops/pull/9074
	# https://github.com/karmab/kcli/commit/b1a8eff658d17cf4e28162f0fa2c8b2b10e5ad00
	SUBSYSTEM=="net", ACTION=="add|change|move", ENV{INTERFACE}=="flannel.1", RUN+="/usr/sbin/ethtool -K flannel.1 tx-checksum-ip-generic off"

It is still unclear why this is needed only for rootful, though.

Any eyes needed here from the Moby networking folks? (I know they're pretty busy currently, but if it's useful I can try ask them if they have time to spare to give it eyes)

Thanks, that would be appreciated.

[AkihiroSuda]

Warning: 7m[WARNING] buildkitd has access to images in "buildkit" namespace by default. If you want to give buildkitd access to the images in "default" namespace, run this command with CONTAINERD_NAMESPACE=default

Irrelevant to the topic.
Should be fixed though.