Slow pod to pod networking in cluster with 10gbe NIC

[benosman]

I'm trying to set up a cluster on some very simple VMs from an independent host (non cloud). I have 3 of them in a control plane with scheduling enabled, and each node has a 10gbe nic.

However, when testing pod to pod networking speed using iperf3, I can only get between 450-600mbps between pods on different nodes. When using kubectl debug, I can get ~9gbps between nodes or between a pod and a node.

I've tested using flannel and cilium, this is a cluster without any workloads and has been created using gen config with the minimal config for test purposes. No firewall, kubespan disabled, the only real config I added is some network interfaces as the cloud init config is in a non standard location.

Install method
AMD64 Metal iso, with system extensions:

• siderolabs/amd-ucode
• siderolabs/iscsi-tools
• siderolabs/qemu-guest-agent
• siderolabs/util-linux-tools

VM Specs
AMD Epyc platform - 4 virtual cores
8GB RAM
10gbe vnic
1 public ip address

What i've tried
Flannel CNI
Cillium CNI, with kube proxy disabled
Experimented with different mtus, 1500, 9000
Added various kernel flags suggested by the host

Since the ip addresses are public, I'd prefer not to share the support zip here, but let me know how I can provide further logs or config info if necessary, or what else I should try.

This is the network config. All nodes are on the same /24 subnet, but it is not private to me.

machine:
  ...
  network:
    hostname: node0
    interfaces:
      - interface: eth0
        mtu: 1500
        addresses:
          - a.b.c.145/24
        routes:
          - network: 0.0.0.0/0
            gateway: a.b.c.1
    nameservers:
      - 9.9.9.9
      - 149.112.112.112

pod on node2# iperf3 -c 10.244.0.4 -p 12345
Connecting to host 10.244.0.4, port 12345
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  60.4 MBytes   506 Mbits/sec    7    508 KBytes       
[  4]   1.00-2.00   sec  58.0 MBytes   486 Mbits/sec    0    516 KBytes       
[  4]   2.00-3.00   sec  57.9 MBytes   486 Mbits/sec    0    522 KBytes       
[  4]   3.00-4.00   sec  57.0 MBytes   478 Mbits/sec    0    385 KBytes       
[  4]   4.00-5.00   sec  58.0 MBytes   487 Mbits/sec    0    451 KBytes       
[  4]   5.00-6.00   sec  58.1 MBytes   488 Mbits/sec    0    500 KBytes       
[  4]   6.00-7.00   sec  58.0 MBytes   486 Mbits/sec    0    371 KBytes  

pod on node2# iperf3 -c node0 -p 12345 
Connecting to host node0, port 12345
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1022 MBytes  8.57 Gbits/sec    2   4.05 MBytes       
[  4]   1.00-2.00   sec  1018 MBytes  8.54 Gbits/sec    0   4.05 MBytes       
[  4]   2.00-3.00   sec  1022 MBytes  8.58 Gbits/sec    0   4.05 MBytes       
[  4]   3.00-4.00   sec   961 MBytes  8.06 Gbits/sec    0   4.05 MBytes       
[  4]   4.00-5.00   sec   855 MBytes  7.17 Gbits/sec    1   4.05 MBytes       
[  4]   5.00-6.00   sec  1.00 GBytes  8.60 Gbits/sec    1   4.05 MBytes       
[  4]   6.00-7.00   sec  1015 MBytes  8.52 Gbits/sec    0   4.05 MBytes  

pod on node2# iperf3 -c node1 -p 12345 
Connecting to host node1 port 12345
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.00 GBytes  8.62 Gbits/sec  154   1.48 MBytes       
[  4]   1.00-2.00   sec   849 MBytes  7.12 Gbits/sec   77   1.85 MBytes       
[  4]   2.00-3.00   sec  1004 MBytes  8.42 Gbits/sec  109   2.21 MBytes       
[  4]   3.00-4.00   sec  1.01 GBytes  8.69 Gbits/sec   95   2.52 MBytes       
[  4]   4.00-5.00   sec  1.02 GBytes  8.73 Gbits/sec   32   2.80 MBytes       
[  4]   5.00-6.00   sec  1.02 GBytes  8.77 Gbits/sec    0   3.07 MBytes 

pod on node2# iperf3 -c node2 -p 12345 
Connecting to host node2 port 12345
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  3.40 GBytes  29.2 Gbits/sec    0   3.61 MBytes       
[  4]   1.00-2.00   sec  2.75 GBytes  23.6 Gbits/sec    1   2.53 MBytes       
[  4]   2.00-3.00   sec  2.31 GBytes  19.9 Gbits/sec    0   3.15 MBytes       
[  4]   3.00-4.00   sec  2.33 GBytes  20.0 Gbits/sec    0   3.15 MBytes  

[smira]

If we try to split this into pieces, Talos Linux as the OS only manages host-to-host traffic, which in your case seems to achieve wire speed (around 10 Gbit/s), so there is no problem at the OS level.

The next layer is the CNI, and in the default setup of both Flannel and Cilium they use vxlan encapsulation to wrap pod-to-pod traffic into host-to-host traffic, and this layer is the one causing performance issues.

I would probably go into the following directions:

1. With Cilium, you could try to use BGP & native routing to completely avoid encapsulation. Or explore any other ways offered by Cilium.
2. You could try to dig deeper into the vxlan encapsulation - e.g. using talosctl pcap, try to capture some traffic on the host interface which is pod-to-pod, and look into the packet size (does it use full MTU, do the packets get fragmeneted? does it use bigger MTU than allowed? any packet drops?).
3. Look into resource consumption (what if you're out of CPU resources and that causes issues?)

[benosman]

Thanks for the reply.

SInce posting, I did start to experiment with cilium config including native routing - not much in the way of gains yet, but I will do a bit more.

I will also try capturing some traffic for inspection. On resource consumption, the VMs are idle and there are no surges in load when running these iperf3 tests.

Finally, I may try an install of k3s on a host supported OS to find out if it has the same issue with pod networking. It may also be easier for me to get some support from them on that OS.

[benosman]

I've looked a bit further into this, and I can reproduce the same issue on k3s with flannel. It seems like vxlan encapsulation isn't working well on my vms.

I will have another try with cilium and native routing, but on that topic, would that work with Kubespan if I wanted to add that in the future?

[smira]

KubeSpan works on host-to-host level, so if you use native routing for pod communication, KubeSpan won't catch that traffic at all.

But I think vxlan degradation is not expected to be that bad. You should have an issue somewhere.

[benosman]

It seems to be an issue with any udp traffic with this particular host being considerably slower than tcp. I've reached out to them but so far they class it as software related.

It's not talos related though, that is for sure.