Talos with multiple nics for public/private networking

[mayasBK]

Hi,
I am running a multiple node sized Proxmox Cluster with Talos on top for k8s. I planned to have a local private network with DHCP for outgoing internet access like pulling images etc.
For incoming traffic, I wanted to attach a second NIC with static IPs. An independent HAProxy instance is routing the traffic to k8s ingress.

• ens18: Subnet with DHCP - allows outgoing traffic and connection between talos nodes, but only for instances on same proxmox machine
• ens19: static IP - allows connection between all talos nodes across multiple proxmox nodes and used for incoming traffic

Is this something I can do with talos without messing up network configuration inside talos? I noticed that NodePorts are not listening on both interfaces, and I do not know how to control which interface is chosen for that. Flannel is showing a different public IP than k8s is reporting for the node.
I also noticed that etcd is picking random IPs for Peering and Client URLs.

If I go this route with 2 NICs I'd like to have some control which is used for what to prevent unforeseen issues along the road. Is there a guide which helps making the correct changes? Or what is the best approach here?

As example for etcd, this seems to picked automaticlly, how do I make sure that it picks the correct interface here? If it picks ens18 with 172.20.0.0/24 for example I would have connection issues as soon as I migrate a talos instance to another proxmox node.

talosctl --talosconfig=talosconfig etcd members
NODE          ID                 HOSTNAME   PEER URLS                 CLIENT URLS               LEARNER
172.20.0.30   1784e7ec7b1f39ad   k8s-cp03   https://10.50.1.43:2380   https://10.50.1.43:2379   false
172.20.0.30   a80e3acd8b4b631b   k8s-cp02   https://10.50.1.42:2380   https://10.50.1.42:2379   false
172.20.0.30   c81fca03d92ab261   k8s-cp01   https://10.50.1.41:2380   https://10.50.1.41:2379   false

Config of current worker node:

apiVersion: v1
kind: Node
metadata:
  annotations:
    flannel.alpha.coreos.com/backend-data: '{"VNI":1,"VtepMAC":"f6:36:7e:2e:89:bc"}'
    flannel.alpha.coreos.com/backend-type: vxlan
    flannel.alpha.coreos.com/kube-subnet-manager: "true"
    flannel.alpha.coreos.com/public-ip: 172.20.0.33
  name: k8s-worker01
spec:
  podCIDR: 10.244.6.0/24
  podCIDRs:
  - 10.244.6.0/24
status:
  addresses:
  - address: 10.50.1.44
    type: InternalIP
  - address: k8s-worker01
    type: Hostname

[smira]

Please see https://www.talos.dev/v1.9/introduction/prodnotes/#multihoming, if you have further questions, please tell us.

Flannel has its own way to pick up the IP unfortunately.

[mayasBK]

@smira thank you. That link was helpfull, exactly what I was looking for.

In the meantime, another issue occurred in that context. Flannel uses vxlan as it turns out, and my Talos nodes are connected via vxlan as well. Vxlan on top of vxlan doesn't work well. Is there a recommendation for that use case as well?

[smira]

Flannel CNI is just one of the available CNIs for Kubernetes, I don't know about specifically vxlan on vxlan.
Also Flannel supports multiple backends.

Talos LInux comes with simple Flannel installation by default, but you can disable that, and install any other CNI, including custom configuration of Flannel. AFAIK by default all CNIs use vxlan, but they have options.

This is not specific to Talos Linux, so you might need to research this and tell us what's the best solution.