load balancer IPs inaccessible on LAN

[nazihahmed]

Hello, so i have followed your doc on deploying to proxmox,
i created a 3 node cluster with default generated settings, installed metallb, tallosctl health looks all good
as well as all pods running cluster wide, i can see my load balancers getting IPs in the range i defined,
however i cannot seem to successfully reach any of these services, i am wondering if there is some security configuration i need to set or change in talos to expose the subnet that i use (10.123.1.0/24)
note: for each node in the cluster i have set a static IP in pfsense, not sure if this affects the controller for getting new addresses that i defined for metallb

$ kubectl expose deploy nginx --port 80 --type LoadBalancer
$ kubectl get svc
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)        AGE
kubernetes   ClusterIP      10.96.0.1       <none>         443/TCP        18h
nginx        LoadBalancer   10.109.38.224   10.123.1.220   80:30044/TCP   11s

$ talosctl get addresses
NODE          NAMESPACE   TYPE            ID                                          VERSION   ADDRESS                        LINK
10.123.1.40   network     AddressStatus   cni0/10.244.0.1/24                          1         10.244.0.1/24                  cni0
10.123.1.40   network     AddressStatus   cni0/fe80::e831:24ff:fee1:17f2/64           2         fe80::e831:24ff:fee1:17f2/64   cni0
10.123.1.40   network     AddressStatus   eth0/10.123.1.40/24                         1         10.123.1.40/24                 eth0
10.123.1.40   network     AddressStatus   eth0/fe80::10d6:cbff:fe94:6bb3/64           2         fe80::10d6:cbff:fe94:6bb3/64   eth0
10.123.1.40   network     AddressStatus   flannel.1/10.244.0.0/32                     1         10.244.0.0/32                  flannel.1
10.123.1.40   network     AddressStatus   flannel.1/fe80::7411:c4ff:fe29:7983/64      2         fe80::7411:c4ff:fe29:7983/64   flannel.1
10.123.1.40   network     AddressStatus   lo/127.0.0.1/8                              1         127.0.0.1/8                    lo
10.123.1.40   network     AddressStatus   lo/::1/128                                  1         ::1/128                        lo
10.123.1.40   network     AddressStatus   veth28256560/fe80::d804:f1ff:fe3a:6a5e/64   2         fe80::d804:f1ff:fe3a:6a5e/64   veth28256560
10.123.1.40   network     AddressStatus   vethd6142e4e/fe80::bc35:dfff:fec5:b9a/64    2         fe80::bc35:dfff:fec5:b9a/64    vethd6142e4e

$ kubectl get all -A -o wide
NAMESPACE        NAME                                        READY   STATUS      RESTARTS        AGE     IP            NODE            NOMINATED NODE   READINESS GATES
default          pod/nginx-748c667d99-rsmsv                  1/1     Running     0               113m    10.244.1.4    talos-node-2    <none>           <none>
kube-system      pod/coredns-5597575654-8v6ml                1/1     Running     0               49m     10.244.3.3    talos-node-1    <none>           <none>
kube-system      pod/coredns-5597575654-lnkqx                1/1     Running     0               49m     10.244.1.5    talos-node-2    <none>           <none>
kube-system      pod/kube-apiserver-talos-a1m-r5w            1/1     Running     0               25m     10.123.1.40   talos-a1m-r5w   <none>           <none>
kube-system      pod/kube-apiserver-talos-ctrlr-0            1/1     Running     0               5m14s   10.123.1.40   talos-ctrlr-0   <none>           <none>
kube-system      pod/kube-controller-manager-talos-a1m-r5w   1/1     Running     1 (26m ago)     25m     10.123.1.40   talos-a1m-r5w   <none>           <none>
kube-system      pod/kube-controller-manager-talos-ctrlr-0   1/1     Running     1 (6m1s ago)    5m14s   10.123.1.40   talos-ctrlr-0   <none>           <none>
kube-system      pod/kube-flannel-4pk2n                      1/1     Running     0               19h     10.123.1.43   talos-node-2    <none>           <none>
kube-system      pod/kube-flannel-d6dgf                      1/1     Running     0               26m     10.123.1.40   talos-a1m-r5w   <none>           <none>
kube-system      pod/kube-flannel-m924d                      1/1     Running     0               19h     10.123.1.40   talos-ctrlr-0   <none>           <none>
kube-system      pod/kube-flannel-w8ff9                      1/1     Running     0               19h     10.123.1.42   talos-node-1    <none>           <none>
kube-system      pod/kube-flannel-zcn5v                      1/1     Running     0               19m     10.123.1.41   talos-node-0    <none>           <none>
kube-system      pod/kube-proxy-bttq7                        1/1     Running     0               19h     10.123.1.42   talos-node-1    <none>           <none>
kube-system      pod/kube-proxy-dvwcx                        1/1     Running     0               26m     10.123.1.40   talos-a1m-r5w   <none>           <none>
kube-system      pod/kube-proxy-jfn26                        1/1     Running     0               19h     10.123.1.43   talos-node-2    <none>           <none>
kube-system      pod/kube-proxy-ngt92                        1/1     Running     0               19h     10.123.1.40   talos-ctrlr-0   <none>           <none>
kube-system      pod/kube-proxy-xbhvr                        1/1     Running     1 (19m ago)     19h     10.123.1.41   talos-node-0    <none>           <none>
kube-system      pod/kube-scheduler-talos-a1m-r5w            1/1     Running     1 (26m ago)     25m     10.123.1.40   talos-a1m-r5w   <none>           <none>
kube-system      pod/kube-scheduler-talos-ctrlr-0            1/1     Running     1 (6m1s ago)    5m14s   10.123.1.40   talos-ctrlr-0   <none>           <none>
metallb-system   pod/controller-577b5bdfcc-g6sht             0/1     Completed   0               135m    10.244.2.2    talos-node-0    <none>           <none>
metallb-system   pod/controller-f44b7ff6d-j2j6r              1/1     Running     0               108s    10.244.2.3    talos-node-0    <none>           <none>
metallb-system   pod/speaker-gk5bt                           1/1     Running     0               135m    10.123.1.42   talos-node-1    <none>           <none>
metallb-system   pod/speaker-gmxbd                           1/1     Running     1 (4m41s ago)   135m    10.123.1.40   talos-ctrlr-0   <none>           <none>
metallb-system   pod/speaker-m2gfs                           1/1     Running     1 (25m ago)     26m     10.123.1.40   talos-a1m-r5w   <none>           <none>
metallb-system   pod/speaker-mpg49                           1/1     Running     0               135m    10.123.1.43   talos-node-2    <none>           <none>
metallb-system   pod/speaker-ntjxp                           1/1     Running     0               19m     10.123.1.41   talos-node-0    <none>           <none>

NAMESPACE        NAME                      TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                  AGE    SELECTOR
default          service/kubernetes        ClusterIP      10.96.0.1       <none>         443/TCP                  19h    <none>
default          service/nginx             LoadBalancer   10.100.79.237   10.123.1.220   80:31063/TCP             65m    app=nginx
kube-system      service/kube-dns          ClusterIP      10.96.0.10      <none>         53/UDP,53/TCP,9153/TCP   19h    k8s-app=kube-dns
metallb-system   service/webhook-service   ClusterIP      10.96.25.166    <none>         443/TCP                  135m   component=controller

NAMESPACE        NAME                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE    CONTAINERS     IMAGES                               SELECTOR
kube-system      daemonset.apps/kube-flannel   5         5         4       5            4           <none>                   19h    kube-flannel   ghcr.io/siderolabs/flannel:v0.20.2   k8s-app=flannel,tier=node
kube-system      daemonset.apps/kube-proxy     5         5         4       5            4           <none>                   19h    kube-proxy     registry.k8s.io/kube-proxy:v1.26.0   k8s-app=kube-proxy,tier=node
metallb-system   daemonset.apps/speaker        4         4         4       4            4           kubernetes.io/os=linux   135m   speaker        quay.io/metallb/speaker:v0.13.7      app=metallb,component=speaker

NAMESPACE        NAME                         READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS   IMAGES                               SELECTOR
default          deployment.apps/nginx        1/1     1            1           113m   nginx        nginx                                app=nginx
kube-system      deployment.apps/coredns      2/2     2            2           19h    coredns      docker.io/coredns/coredns:1.10.0     k8s-app=kube-dns
metallb-system   deployment.apps/controller   1/1     1            1           135m   controller   quay.io/metallb/controller:v0.13.7   app=metallb,component=controller

NAMESPACE        NAME                                    DESIRED   CURRENT   READY   AGE    CONTAINERS   IMAGES                               SELECTOR
default          replicaset.apps/nginx-748c667d99        1         1         1       113m   nginx        nginx                                app=nginx,pod-template-hash=748c667d99
kube-system      replicaset.apps/coredns-5597575654      2         2         2       19h    coredns      docker.io/coredns/coredns:1.10.0     k8s-app=kube-dns,pod-template-hash=5597575654
metallb-system   replicaset.apps/controller-577b5bdfcc   0         0         0       135m   controller   quay.io/metallb/controller:v0.13.7   app=metallb,component=controller,pod-template-hash=577b5bdfcc
metallb-system   replicaset.apps/controller-f44b7ff6d    1         1         1       109s   controller   quay.io/metallb/controller:v0.13.7   app=metallb,component=controller,pod-template-hash=f44b7ff6d

there are very few people out there sharing anything on talos so this is my only resource to get help
I appreciate any advice!
thanks.

[smira]

I'm not an expert in MetalLB, but the fact that the address for the LoadBalancer 10.123.1.220 wasn't assigned to the node (talosctl get addresses) by MetaLB, sounds like it is the root cause.

I would look into MetalLB logs, and see if it got assigned to a different node (?).

[steverfrancis]

You may also want to ask in our Slack, if you have not yet.

…

On Mon, Dec 26, 2022 at 2:46 AM Andrey Smirnov ***@***.***> wrote: I'm not an expert in MetalLB, but the fact that the address for the LoadBalancer 10.123.1.220 wasn't assigned to the node (talosctl get addresses) by MetaLB, sounds like it is the root cause. I would look into MetalLB logs, and see if it got assigned to a different node (?). — Reply to this email directly, view it on GitHub <#6686 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQGWG5M2OZT2VMKGKYKCYKLWPFZQLANCNFSM6AAAAAATJQZZFM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[ak47rwd]

收到！