Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spidernet EgressGateway Fails Due to iptables-nft Compatibility Issue #1622

Open
psavva opened this issue Feb 12, 2025 · 7 comments
Open

Spidernet EgressGateway Fails Due to iptables-nft Compatibility Issue #1622

psavva opened this issue Feb 12, 2025 · 7 comments
Assignees
@lou-lan

Comments

@psavva
Copy link

psavva commented Feb 12, 2025

Describe the Version

Kubernetes Distribution: K3s Rancher
CNI Plugin: Flannel (not using Spidernet CNI, only Egress)
Spidernet EgressGateway Version: latest
iptables Version: iptables v1.8.10 (nf_tables)
OS and Kernel Version: Linux datapulse-cpx31-master3 6.8.0-52-generic #53-Ubuntu SMP PREEMPT_DYNAMIC Sat Jan 11 00:06:25 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Describe the Bug
When deploying Spidernet EgressGateway, the agent fails due to iptables-nft incompatibility. The logs show errors indicating that iptables-save cannot process the filter table due to existing nftables rules.

How To Reproduce

Deploy K3s Rancher with Flannel as the CNI.
Install EgressGateway using Helm:
helm repo add egressgateway https://spidernet-io.github.io/egressgateway/
helm repo update
helm install egressgateway egressgateway/egressgateway -n kube-system --set feature.tunnelIpv4Subnet="192.200.0.1/16" --wait --debug
Check the EgressGateway agent logs:
kubectl logs -n kube-system -l app.kubernetes.io/name=egressgateway-agent
Expected Behavior
EgressGateway should successfully apply iptables rules and function as expected.

Screenshots and Log
Relevant error logs:
level="error" ts="2025-02-12T05:17:23.035Z" caller="iptables/table.go:679" msg="" table="filter" ipVersion=4 line="# Table `filter' is incompatible, use 'nft' tool."
error="iptables-save failed because there are incompatible nft rules in the table, remove the nft rules to continue"

root@datapulse-cpx31-master3:~# kubectl logs egressgateway-agent-vwrzf -n kube-system
{"NodeName":"datapulse-cpx31-master3","LeaderElection":true,"LeaderElectionNamespace":"","LeaderElectionID":"egressgateway","LeaderElectionLostRestart":false,"MetricsBindAddress":"0","HealthProbeBindAddress":":5810","GopsPort":5812,"WebhookPort":8881,"PyroscopeServerAddr":"","PodName":"egressgateway-agent-vwrzf","PodNamespace":"kube-system","GolangMaxProcs":-1,"TLSCertDir":"/etc/tls","ConfigMapPath":"/tmp/config-map/conf.yml","UseDevMode":true,"Level":"info","WithCaller":true,"Encoder":"json","FileConfig":{"EnableIPv4":true,"EnableIPv6":false,"IPTables":{"BackendMode":"nft","RefreshIntervalSecond":90,"PostWriteIntervalSecond":1,"LockTimeoutSecond":0,"LockProbeIntervalMillis":50,"InitialPostWriteIntervalSecond":0,"RestoreSupportsLock":true,"LockFilePath":"/run/xtables.lock"},"DatapathMode":"iptables","TunnelIpv4Subnet":"172.31.0.0/16","TunnelIpv6Subnet":"fd11::/112","TunnelDetectMethod":"interface=eth0","VXLAN":{"Name":"egress.vxlan","ID":100,"Port":7789,"DisableChecksumOffload":false},"MaxNumberEndpointPerSlice":100,"Mark":"0x26000000","AnnouncedInterfacesToExclude":["^cali.*","br-*"],"EnableGatewayReplyRoute":false,"GatewayReplyRouteTable":600,"GatewayReplyRouteMark":39,"GatewayFailover":{"Enable":false,"TunnelMonitorPeriod":5,"TunnelUpdatePeriod":5,"EipEvictionTimeout":15}}}
{"level":"info","ts":"2025-02-12T05:20:12.746Z","caller":"agent/police.go:1142","msg":"iptables-restore has built-in lock implementation"}
{"level":"info","ts":"2025-02-12T05:20:12.747Z","caller":"iptables/table.go:226","msg":"postWriteInterval too small, defaulting","setValue":"0s","default":"50ms"}
{"level":"info","ts":"2025-02-12T05:20:12.747Z","caller":"iptables/table.go:292","msg":"Enabling iptables-in-nftables-mode workarounds."}
{"level":"info","ts":"2025-02-12T05:20:12.747Z","caller":"iptables/table.go:226","msg":"postWriteInterval too small, defaulting","setValue":"0s","default":"50ms"}
{"level":"info","ts":"2025-02-12T05:20:12.747Z","caller":"iptables/table.go:292","msg":"Enabling iptables-in-nftables-mode workarounds."}
{"level":"info","ts":"2025-02-12T05:20:12.747Z","caller":"iptables/table.go:226","msg":"postWriteInterval too small, defaulting","setValue":"0s","default":"50ms"}
{"level":"info","ts":"2025-02-12T05:20:12.747Z","caller":"iptables/table.go:292","msg":"Enabling iptables-in-nftables-mode workarounds."}
{"level":"info","ts":"2025-02-12T05:20:12.747Z","caller":"manager/server.go:83","msg":"starting server","name":"health probe","addr":"[::]:5810"}
{"level":"info","ts":"2025-02-12T05:20:12.747Z","caller":"controller/controller.go:173","msg":"Starting EventSource","controller":"vxlan","source":"kind source: *v1beta1.EgressTunnel"}
{"level":"info","ts":"2025-02-12T05:20:12.747Z","caller":"controller/controller.go:173","msg":"Starting EventSource","controller":"vxlan","source":"kind source: *v1beta1.EgressGateway"}
{"level":"info","ts":"2025-02-12T05:20:12.747Z","caller":"controller/controller.go:173","msg":"Starting EventSource","controller":"vxlan","source":"kind source: *v1beta1.EgressEndpointSlice"}
{"level":"info","ts":"2025-02-12T05:20:12.747Z","caller":"controller/controller.go:173","msg":"Starting EventSource","controller":"vxlan","source":"kind source: *v1beta1.EgressClusterEndpointSlice"}
{"level":"info","ts":"2025-02-12T05:20:12.747Z","caller":"controller/controller.go:181","msg":"Starting Controller","controller":"vxlan"}
{"level":"info","ts":"2025-02-12T05:20:12.747Z","caller":"controller/controller.go:173","msg":"Starting EventSource","controller":"eip","source":"kind source: *v1beta1.EgressPolicy"}
{"level":"info","ts":"2025-02-12T05:20:12.747Z","caller":"controller/controller.go:173","msg":"Starting EventSource","controller":"eip","source":"kind source: *v1beta1.EgressClusterPolicy"}
{"level":"info","ts":"2025-02-12T05:20:12.747Z","caller":"controller/controller.go:181","msg":"Starting Controller","controller":"eip"}
{"level":"info","ts":"2025-02-12T05:20:12.747Z","caller":"controller/controller.go:173","msg":"Starting EventSource","controller":"policy","source":"kind source: *v1beta1.EgressGateway"}
{"level":"info","ts":"2025-02-12T05:20:12.747Z","caller":"controller/controller.go:173","msg":"Starting EventSource","controller":"policy","source":"kind source: *v1beta1.EgressPolicy"}
{"level":"info","ts":"2025-02-12T05:20:12.747Z","caller":"controller/controller.go:173","msg":"Starting EventSource","controller":"policy","source":"kind source: *v1beta1.EgressClusterPolicy"}
{"level":"info","ts":"2025-02-12T05:20:12.747Z","caller":"controller/controller.go:173","msg":"Starting EventSource","controller":"policy","source":"kind source: *v1beta1.EgressEndpointSlice"}
{"level":"info","ts":"2025-02-12T05:20:12.747Z","caller":"controller/controller.go:173","msg":"Starting EventSource","controller":"policy","source":"kind source: *v1beta1.EgressClusterEndpointSlice"}
{"level":"info","ts":"2025-02-12T05:20:12.747Z","caller":"controller/controller.go:173","msg":"Starting EventSource","controller":"policy","source":"kind source: *v1beta1.EgressClusterInfo"}
{"level":"info","ts":"2025-02-12T05:20:12.747Z","caller":"controller/controller.go:173","msg":"Starting EventSource","controller":"policy","source":"kind source: *v1beta1.EgressTunnel"}
{"level":"info","ts":"2025-02-12T05:20:12.747Z","caller":"controller/controller.go:181","msg":"Starting Controller","controller":"policy"}
{"level":"info","ts":"2025-02-12T05:20:12.748Z","caller":"profiling/manager.go:34","msg":"gops is started","addr":":5812"}
{"level":"info","ts":"2025-02-12T05:20:12.748Z","caller":"layer2/announcer.go:139","msg":"created ARP responder for interface","interface":"eth0","event":"createARPResponder"}
{"level":"info","ts":"2025-02-12T05:20:12.748Z","caller":"layer2/announcer.go:148","msg":"created NDP responder for interface","interface":"eth0","event":"createNDPResponder"}
{"level":"info","ts":"2025-02-12T05:20:12.749Z","caller":"layer2/announcer.go:139","msg":"created ARP responder for interface","interface":"enp7s0","event":"createARPResponder"}
{"level":"info","ts":"2025-02-12T05:20:12.749Z","caller":"layer2/announcer.go:148","msg":"created NDP responder for interface","interface":"enp7s0","event":"createNDPResponder"}
{"level":"info","ts":"2025-02-12T05:20:12.749Z","caller":"layer2/announcer.go:139","msg":"created ARP responder for interface","interface":"flannel.1","event":"createARPResponder"}
{"level":"info","ts":"2025-02-12T05:20:12.749Z","caller":"layer2/announcer.go:148","msg":"created NDP responder for interface","interface":"flannel.1","event":"createNDPResponder"}
{"level":"info","ts":"2025-02-12T05:20:12.750Z","caller":"layer2/announcer.go:139","msg":"created ARP responder for interface","interface":"cni0","event":"createARPResponder"}
{"level":"info","ts":"2025-02-12T05:20:12.750Z","caller":"layer2/announcer.go:148","msg":"created NDP responder for interface","interface":"cni0","event":"createNDPResponder"}
{"level":"info","ts":"2025-02-12T05:20:12.753Z","caller":"layer2/announcer.go:139","msg":"created ARP responder for interface","interface":"egress.vxlan","event":"createARPResponder"}
{"level":"info","ts":"2025-02-12T05:20:12.753Z","caller":"layer2/announcer.go:148","msg":"created NDP responder for interface","interface":"egress.vxlan","event":"createNDPResponder"}
{"level":"info","ts":"2025-02-12T05:20:12.856Z","caller":"controller/controller.go:215","msg":"Starting workers","controller":"vxlan","worker count":1}
{"level":"info","ts":"2025-02-12T05:20:12.856Z","caller":"agent/vxlan.go:67","msg":"first reconcile of egresstunnel agent, init TunnelPeerMap"}
{"level":"info","ts":"2025-02-12T05:20:12.856Z","caller":"agent/vxlan.go:78","msg":"reconciling","name":"datapulse-cx42-pool-autoscaled-cx42-worker1","kind":"EgressTunnel"}
{"level":"info","ts":"2025-02-12T05:20:12.856Z","caller":"agent/vxlan.go:152","msg":"parent ip not ready, skip","name":"datapulse-cx42-pool-autoscaled-cx42-worker1","kind":"EgressTunnel","peer":"datapulse-cx42-pool-autoscaled-cx42-worker1"}
{"level":"info","ts":"2025-02-12T05:20:12.856Z","caller":"agent/vxlan.go:78","msg":"reconciling","name":"datapulse-cx42-pool-autoscaled-cx42-worker3","kind":"EgressTunnel"}
{"level":"info","ts":"2025-02-12T05:20:12.856Z","caller":"agent/vxlan.go:152","msg":"parent ip not ready, skip","name":"datapulse-cx42-pool-autoscaled-cx42-worker3","kind":"EgressTunnel","peer":"datapulse-cx42-pool-autoscaled-cx42-worker3"}
{"level":"info","ts":"2025-02-12T05:20:12.856Z","caller":"agent/vxlan.go:78","msg":"reconciling","name":"autoscaled-cx42-55e449771fadb66b","kind":"EgressTunnel"}
{"level":"info","ts":"2025-02-12T05:20:12.856Z","caller":"agent/vxlan.go:152","msg":"parent ip not ready, skip","name":"autoscaled-cx42-55e449771fadb66b","kind":"EgressTunnel","peer":"autoscaled-cx42-55e449771fadb66b"}
{"level":"info","ts":"2025-02-12T05:20:12.856Z","caller":"agent/vxlan.go:78","msg":"reconciling","name":"autoscaled-cx42-642b0b25a5ad7476","kind":"EgressTunnel"}
{"level":"info","ts":"2025-02-12T05:20:12.856Z","caller":"agent/vxlan.go:152","msg":"parent ip not ready, skip","name":"autoscaled-cx42-642b0b25a5ad7476","kind":"EgressTunnel","peer":"autoscaled-cx42-642b0b25a5ad7476"}
{"level":"info","ts":"2025-02-12T05:20:12.856Z","caller":"agent/vxlan.go:78","msg":"reconciling","name":"datapulse-cpx31-master1","kind":"EgressTunnel"}
{"level":"info","ts":"2025-02-12T05:20:12.857Z","caller":"agent/vxlan.go:78","msg":"reconciling","name":"datapulse-cpx31-master2","kind":"EgressTunnel"}
{"level":"info","ts":"2025-02-12T05:20:12.858Z","caller":"agent/vxlan.go:78","msg":"reconciling","name":"datapulse-cpx31-master3","kind":"EgressTunnel"}
{"level":"info","ts":"2025-02-12T05:20:12.858Z","caller":"agent/vxlan.go:78","msg":"reconciling","name":"default","kind":"EgressGateway"}
{"level":"info","ts":"2025-02-12T05:20:12.907Z","caller":"controller/controller.go:215","msg":"Starting workers","controller":"policy","worker count":1}
{"level":"info","ts":"2025-02-12T05:20:12.907Z","caller":"controller/controller.go:215","msg":"Starting workers","controller":"eip","worker count":1}
{"level":"info","ts":"2025-02-12T05:20:12.908Z","caller":"agent/police.go:63","msg":"starting first reconciliation of policy controller"}
{"level":"info","ts":"2025-02-12T05:20:12.908Z","caller":"agent/police.go:114","msg":"apply policy"}
{"level":"info","ts":"2025-02-12T05:20:12.924Z","caller":"iptables/table.go:350","msg":"queueing update of chain.","table":"mangle","ipVersion":4,"chainName":"EGRESSGATEWAY-REPLY-ROUTING"}
{"level":"info","ts":"2025-02-12T05:20:12.924Z","caller":"iptables/table.go:350","msg":"queueing update of chain.","table":"mangle","ipVersion":4,"chainName":"EGRESSGATEWAY-MARK-REQUEST"}
{"level":"info","ts":"2025-02-12T05:20:12.924Z","caller":"iptables/table.go:424","msg":"chain became referenced, marking it for programming","table":"mangle","ipVersion":4,"chainName":"EGRESSGATEWAY-MARK-REQUEST"}
{"level":"info","ts":"2025-02-12T05:20:12.924Z","caller":"iptables/table.go:424","msg":"chain became referenced, marking it for programming","table":"mangle","ipVersion":4,"chainName":"EGRESSGATEWAY-REPLY-ROUTING"}
{"level":"info","ts":"2025-02-12T05:20:12.924Z","caller":"iptables/table.go:350","msg":"queueing update of chain.","table":"mangle","ipVersion":4,"chainName":"EGRESSGATEWAY-MARK-REQUEST"}
{"level":"info","ts":"2025-02-12T05:20:12.924Z","caller":"iptables/table.go:350","msg":"queueing update of chain.","table":"mangle","ipVersion":4,"chainName":"EGRESSGATEWAY-REPLY-ROUTING"}
{"level":"info","ts":"2025-02-12T05:20:12.924Z","caller":"iptables/table.go:350","msg":"queueing update of chain.","table":"nat","ipVersion":4,"chainName":"EGRESSGATEWAY-SNAT-EIP"}
{"level":"info","ts":"2025-02-12T05:20:12.924Z","caller":"iptables/table.go:424","msg":"chain became referenced, marking it for programming","table":"nat","ipVersion":4,"chainName":"EGRESSGATEWAY-SNAT-EIP"}
{"level":"info","ts":"2025-02-12T05:20:12.924Z","caller":"iptables/table.go:779","msg":"updating post-write interval","table":"nat","ipVersion":4,"newPostWriteInterval":"100ms"}
{"level":"info","ts":"2025-02-12T05:20:12.924Z","caller":"iptables/table.go:779","msg":"updating post-write interval","table":"nat","ipVersion":4,"newPostWriteInterval":"200ms"}
{"level":"info","ts":"2025-02-12T05:20:13.124Z","caller":"iptables/table.go:779","msg":"updating post-write interval","table":"filter","ipVersion":4,"newPostWriteInterval":"100ms"}
{"level":"info","ts":"2025-02-12T05:20:13.124Z","caller":"iptables/table.go:779","msg":"updating post-write interval","table":"filter","ipVersion":4,"newPostWriteInterval":"200ms"}
{"level":"info","ts":"2025-02-12T05:20:13.124Z","caller":"iptables/table.go:779","msg":"updating post-write interval","table":"filter","ipVersion":4,"newPostWriteInterval":"400ms"}
{"level":"error","ts":"2025-02-12T05:20:13.147Z","caller":"iptables/table.go:679","msg":"","table":"filter","ipVersion":4,"line":"# Table `filter' is incompatible, use 'nft' tool.","error":"iptables-save failed because there are incompatible nft rules in the table, remove the nft rules to continue","stacktrace":"github.com/spidernet-io/egressgateway/pkg/iptables.(*Table).readHashesAndRulesFrom\n\t/src/pkg/iptables/table.go:679\ngithub.com/spidernet-io/egressgateway/pkg/iptables.(*Table).attemptToGetHashesAndRulesFromDataplane\n\t/src/pkg/iptables/table.go:628\ngithub.com/spidernet-io/egressgateway/pkg/iptables.(*Table).getHashesAndRulesFromDataplane\n\t/src/pkg/iptables/table.go:588\ngithub.com/spidernet-io/egressgateway/pkg/iptables.(*Table).loadDataplaneState\n\t/src/pkg/iptables/table.go:446\ngithub.com/spidernet-io/egressgateway/pkg/iptables.(*Table).Apply\n\t/src/pkg/iptables/table.go:799\ngithub.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).initApplyPolicy\n\t/src/pkg/agent/police.go:310\ngithub.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).Reconcile.func1\n\t/src/pkg/agent/police.go:65\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:74\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).Reconcile\n\t/src/pkg/agent/police.go:62\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:222"}
{"level":"info","ts":"2025-02-12T05:20:13.147Z","caller":"iptables/table.go:632","msg":"killing process after a failure","table":"filter","ipVersion":4,"cmd":"iptables-nft-save","warn":"iptables-save failed because there are incompatible nft rules in the table"}
{"level":"error","ts":"2025-02-12T05:20:13.206Z","caller":"iptables/table.go:591","msg":"failed to run command","table":"filter","ipVersion":4,"command":"iptables-nft-save","error":"iptables-save failed because there are incompatible nft rules in the table","stacktrace":"github.com/spidernet-io/egressgateway/pkg/iptables.(*Table).getHashesAndRulesFromDataplane\n\t/src/pkg/iptables/table.go:591\ngithub.com/spidernet-io/egressgateway/pkg/iptables.(*Table).loadDataplaneState\n\t/src/pkg/iptables/table.go:446\ngithub.com/spidernet-io/egressgateway/pkg/iptables.(*Table).Apply\n\t/src/pkg/iptables/table.go:799\ngithub.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).initApplyPolicy\n\t/src/pkg/agent/police.go:310\ngithub.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).Reconcile.func1\n\t/src/pkg/agent/police.go:65\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:74\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).Reconcile\n\t/src/pkg/agent/police.go:62\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:222"}
{"level":"error","ts":"2025-02-12T05:20:13.342Z","caller":"iptables/table.go:679","msg":"","table":"filter","ipVersion":4,"line":"# Table `filter' is incompatible, use 'nft' tool.","error":"iptables-save failed because there are incompatible nft rules in the table, remove the nft rules to continue","stacktrace":"github.com/spidernet-io/egressgateway/pkg/iptables.(*Table).readHashesAndRulesFrom\n\t/src/pkg/iptables/table.go:679\ngithub.com/spidernet-io/egressgateway/pkg/iptables.(*Table).attemptToGetHashesAndRulesFromDataplane\n\t/src/pkg/iptables/table.go:628\ngithub.com/spidernet-io/egressgateway/pkg/iptables.(*Table).getHashesAndRulesFromDataplane\n\t/src/pkg/iptables/table.go:588\ngithub.com/spidernet-io/egressgateway/pkg/iptables.(*Table).loadDataplaneState\n\t/src/pkg/iptables/table.go:446\ngithub.com/spidernet-io/egressgateway/pkg/iptables.(*Table).Apply\n\t/src/pkg/iptables/table.go:799\ngithub.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).initApplyPolicy\n\t/src/pkg/agent/police.go:310\ngithub.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).Reconcile.func1\n\t/src/pkg/agent/police.go:65\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:74\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).Reconcile\n\t/src/pkg/agent/police.go:62\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:222"}
{"level":"info","ts":"2025-02-12T05:20:13.342Z","caller":"iptables/table.go:632","msg":"killing process after a failure","table":"filter","ipVersion":4,"cmd":"iptables-nft-save","warn":"iptables-save failed because there are incompatible nft rules in the table"}
{"level":"error","ts":"2025-02-12T05:20:13.343Z","caller":"iptables/table.go:591","msg":"failed to run command","table":"filter","ipVersion":4,"command":"iptables-nft-save","error":"iptables-save failed because there are incompatible nft rules in the table","stacktrace":"github.com/spidernet-io/egressgateway/pkg/iptables.(*Table).getHashesAndRulesFromDataplane\n\t/src/pkg/iptables/table.go:591\ngithub.com/spidernet-io/egressgateway/pkg/iptables.(*Table).loadDataplaneState\n\t/src/pkg/iptables/table.go:446\ngithub.com/spidernet-io/egressgateway/pkg/iptables.(*Table).Apply\n\t/src/pkg/iptables/table.go:799\ngithub.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).initApplyPolicy\n\t/src/pkg/agent/police.go:310\ngithub.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).Reconcile.func1\n\t/src/pkg/agent/police.go:65\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:74\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).Reconcile\n\t/src/pkg/agent/police.go:62\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:222"}
{"level":"error","ts":"2025-02-12T05:20:13.583Z","caller":"iptables/table.go:679","msg":"","table":"filter","ipVersion":4,"line":"# Table `filter' is incompatible, use 'nft' tool.","error":"iptables-save failed because there are incompatible nft rules in the table, remove the nft rules to continue","stacktrace":"github.com/spidernet-io/egressgateway/pkg/iptables.(*Table).readHashesAndRulesFrom\n\t/src/pkg/iptables/table.go:679\ngithub.com/spidernet-io/egressgateway/pkg/iptables.(*Table).attemptToGetHashesAndRulesFromDataplane\n\t/src/pkg/iptables/table.go:628\ngithub.com/spidernet-io/egressgateway/pkg/iptables.(*Table).getHashesAndRulesFromDataplane\n\t/src/pkg/iptables/table.go:588\ngithub.com/spidernet-io/egressgateway/pkg/iptables.(*Table).loadDataplaneState\n\t/src/pkg/iptables/table.go:446\ngithub.com/spidernet-io/egressgateway/pkg/iptables.(*Table).Apply\n\t/src/pkg/iptables/table.go:799\ngithub.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).initApplyPolicy\n\t/src/pkg/agent/police.go:310\ngithub.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).Reconcile.func1\n\t/src/pkg/agent/police.go:65\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:74\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).Reconcile\n\t/src/pkg/agent/police.go:62\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:222"}
{"level":"info","ts":"2025-02-12T05:20:13.583Z","caller":"iptables/table.go:632","msg":"killing process after a failure","table":"filter","ipVersion":4,"cmd":"iptables-nft-save","warn":"iptables-save failed because there are incompatible nft rules in the table"}
{"level":"error","ts":"2025-02-12T05:20:13.583Z","caller":"iptables/table.go:591","msg":"failed to run command","table":"filter","ipVersion":4,"command":"iptables-nft-save","error":"iptables-save failed because there are incompatible nft rules in the table","stacktrace":"github.com/spidernet-io/egressgateway/pkg/iptables.(*Table).getHashesAndRulesFromDataplane\n\t/src/pkg/iptables/table.go:591\ngithub.com/spidernet-io/egressgateway/pkg/iptables.(*Table).loadDataplaneState\n\t/src/pkg/iptables/table.go:446\ngithub.com/spidernet-io/egressgateway/pkg/iptables.(*Table).Apply\n\t/src/pkg/iptables/table.go:799\ngithub.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).initApplyPolicy\n\t/src/pkg/agent/police.go:310\ngithub.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).Reconcile.func1\n\t/src/pkg/agent/police.go:65\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:74\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).Reconcile\n\t/src/pkg/agent/police.go:62\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:222"}
{"level":"info","ts":"2025-02-12T05:20:13.750Z","caller":"agent/vxlan.go:480","msg":"vxlan and route has completed"}
{"level":"error","ts":"2025-02-12T05:20:14.005Z","caller":"iptables/table.go:679","msg":"","table":"filter","ipVersion":4,"line":"# Table `filter' is incompatible, use 'nft' tool.","error":"iptables-save failed because there are incompatible nft rules in the table, remove the nft rules to continue","stacktrace":"github.com/spidernet-io/egressgateway/pkg/iptables.(*Table).readHashesAndRulesFrom\n\t/src/pkg/iptables/table.go:679\ngithub.com/spidernet-io/egressgateway/pkg/iptables.(*Table).attemptToGetHashesAndRulesFromDataplane\n\t/src/pkg/iptables/table.go:628\ngithub.com/spidernet-io/egressgateway/pkg/iptables.(*Table).getHashesAndRulesFromDataplane\n\t/src/pkg/iptables/table.go:588\ngithub.com/spidernet-io/egressgateway/pkg/iptables.(*Table).loadDataplaneState\n\t/src/pkg/iptables/table.go:446\ngithub.com/spidernet-io/egressgateway/pkg/iptables.(*Table).Apply\n\t/src/pkg/iptables/table.go:799\ngithub.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).initApplyPolicy\n\t/src/pkg/agent/police.go:310\ngithub.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).Reconcile.func1\n\t/src/pkg/agent/police.go:65\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:74\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).Reconcile\n\t/src/pkg/agent/police.go:62\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:222"}
{"level":"info","ts":"2025-02-12T05:20:14.005Z","caller":"iptables/table.go:632","msg":"killing process after a failure","table":"filter","ipVersion":4,"cmd":"iptables-nft-save","warn":"iptables-save failed because there are incompatible nft rules in the table"}
{"level":"error","ts":"2025-02-12T05:20:14.006Z","caller":"iptables/table.go:591","msg":"failed to run command","table":"filter","ipVersion":4,"command":"iptables-nft-save","error":"iptables-save failed because there are incompatible nft rules in the table","stacktrace":"github.com/spidernet-io/egressgateway/pkg/iptables.(*Table).getHashesAndRulesFromDataplane\n\t/src/pkg/iptables/table.go:591\ngithub.com/spidernet-io/egressgateway/pkg/iptables.(*Table).loadDataplaneState\n\t/src/pkg/iptables/table.go:446\ngithub.com/spidernet-io/egressgateway/pkg/iptables.(*Table).Apply\n\t/src/pkg/iptables/table.go:799\ngithub.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).initApplyPolicy\n\t/src/pkg/agent/police.go:310\ngithub.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).Reconcile.func1\n\t/src/pkg/agent/police.go:65\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:74\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).Reconcile\n\t/src/pkg/agent/police.go:62\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:222"}
{"level":"info","ts":"2025-02-12T05:20:14.006Z","caller":"controller/controller.go:110","msg":"Observed a panic in reconciler: iptables-nft-save command failed after retries","controller":"policy","object":{"name":"default","namespace":"EgressGateway/"},"namespace":"EgressGateway/","name":"default","reconcileID":"81062cd0-e18b-453f-a27f-ec9633ce361d"}
panic: iptables-nft-save command failed after retries [recovered]
        panic: iptables-nft-save command failed after retries

goroutine 273 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
        /src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:111 +0x1e5
panic({0x18326a0?, 0xc000273e30?})
        /usr/local/go/src/runtime/panic.go:770 +0x132
github.com/spidernet-io/egressgateway/pkg/iptables.(*Table).getHashesAndRulesFromDataplane(0xc000449340)
        /src/pkg/iptables/table.go:597 +0x1f4
github.com/spidernet-io/egressgateway/pkg/iptables.(*Table).loadDataplaneState(0xc000449340)
        /src/pkg/iptables/table.go:446 +0xc5
github.com/spidernet-io/egressgateway/pkg/iptables.(*Table).Apply(0xc000449340)
        /src/pkg/iptables/table.go:799 +0x2cc
github.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).initApplyPolicy(0xc000190240)
        /src/pkg/agent/police.go:310 +0xe85
github.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).Reconcile.func1()
        /src/pkg/agent/police.go:65 +0x8f
sync.(*Once).doSlow(0x419f65?, 0xc000487f80?)
        /usr/local/go/src/sync/once.go:74 +0xc2
sync.(*Once).Do(...)
        /usr/local/go/src/sync/once.go:65
github.com/spidernet-io/egressgateway/pkg/agent.(*policeReconciler).Reconcile(0xc000190240, {0x1e04de8, 0xc0001e2c90}, {{{0xc0000135a0?, 0x5?}, {0xc000295336?, 0xc000476d10?}}})
        /src/pkg/agent/police.go:62 +0x99
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1e09d98?, {0x1e04de8?, 0xc0001e2c90?}, {{{0xc0000135a0?, 0xb?}, {0xc000295336?, 0x0?}}})
        /src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00049e420, {0x1e04e20, 0xc000164fa0}, {0x1981680, 0xc00061eba0})
        /src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311 +0x3bc
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00049e420, {0x1e04e20, 0xc000164fa0})
        /src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:261 +0x1be
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
        /src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:222 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 80
        /src/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:218 +0x486
@lou-lan
Copy link
Collaborator

lou-lan commented Feb 14, 2025 •
edited
Loading

iptables-save failed because there are incompatible nft rules in the table, remove the nft rules to continue

I'm installing a new Ubuntu system for testing, and I suspect that nft is enabled on the host instead of iptables-nft. The current egressgateway does not support nft rules yet. Based on the error, it seems that the host is using nft.

@psavva
Copy link
Author

psavva commented Feb 14, 2025 •
edited
Loading

Hi @lou-lan

$:~# iptables --version
iptables v1.8.10 (nf_tables)

$:~# update-alternatives --display iptables
iptables - auto mode
  link best version is /usr/sbin/iptables-nft
  link currently points to /usr/sbin/iptables-nft
  link iptables is /usr/sbin/iptables
  slave iptables-restore is /usr/sbin/iptables-restore
  slave iptables-save is /usr/sbin/iptables-save
/usr/sbin/iptables-legacy - priority 10
  slave iptables-restore: /usr/sbin/iptables-legacy-restore
  slave iptables-save: /usr/sbin/iptables-legacy-save
/usr/sbin/iptables-nft - priority 20
  slave iptables-restore: /usr/sbin/iptables-nft-restore
  slave iptables-save: /usr/sbin/iptables-nft-save

```root@datapulse-cpx31-master1:~# 

I understand that it's actually iptables-nft wich is actually used...

$:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 24.04.1 LTS
Release: 24.04
Codename: noble

@lou-lan
Copy link
Collaborator

lou-lan commented Feb 14, 2025

@psavva

Here should be 3 modes in linux:

  • iptables-legacy (currently supported by egressgateway)
  • iptables-nft (currently supported by egressgateway)
  • nft tables (currently not supported by egressgateway)

Could you please run iptables-save and show me the output in you OS ?

@psavva
Copy link
Author

psavva commented Feb 14, 2025

iptables.log

Please find the log attached

@lou-lan
Copy link
Collaborator

lou-lan commented Feb 18, 2025 •
edited
Loading

@psavva

Hi, in my test, it works for me. I'm using the https://github.com/flannel-io/flannel CNI plugin.

My environment details:

kubectl  version
Client Version: v1.32.2
Kustomize Version: v5.5.0
Server Version: v1.32.2

node@node1:~$ uname -a
Linux node1 6.8.0-53-generic #55-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 17 15:02:14 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux
iptables v1.8.10 (nf_tables)

node@node1:~$ kubectl get pods -o wide -A
NAMESPACE      NAME                                        READY   STATUS    RESTARTS        AGE     IP              NODE    NOMINATED NODE   READINESS GATES
default        visitor-598cbc9646-bjl5j                    1/1     Running   0               6h45m   10.244.0.4      node1   <none>           <none>
kube-flannel   kube-flannel-ds-8d2vt                       1/1     Running   0               7h16m   172.16.25.140   node1   <none>           <none>
kube-flannel   kube-flannel-ds-htjjq                       1/1     Running   0               7h      172.16.25.141   node2   <none>           <none>
kube-system    coredns-668d6bf9bc-7ch26                    1/1     Running   0               7h16m   10.244.0.3      node1   <none>           <none>
kube-system    coredns-668d6bf9bc-qh62l                    1/1     Running   0               7h16m   10.244.0.2      node1   <none>           <none>
kube-system    egressgateway-agent-c77dq                   1/1     Running   0               11m     172.16.25.140   node1   <none>           <none>
kube-system    egressgateway-agent-zh622                   1/1     Running   0               11m     172.16.25.141   node2   <none>           <none>
kube-system    egressgateway-controller-8685fb6b4f-p5t4z   1/1     Running   0               6h50m   10.244.1.2      node2   <none>           <none>
kube-system    etcd-node1                                  1/1     Running   1 (7h17m ago)   7h16m   172.16.25.140   node1   <none>           <none>
kube-system    kube-apiserver-node1                        1/1     Running   1               7h16m   172.16.25.140   node1   <none>           <none>
kube-system    kube-controller-manager-node1               1/1     Running   1 (6h57m ago)   7h16m   172.16.25.140   node1   <none>           <none>
kube-system    kube-proxy-cgg48                            1/1     Running   0               7h16m   172.16.25.140   node1   <none>           <none>
kube-system    kube-proxy-swhhr                            1/1     Running   0               7h      172.16.25.141   node2   <none>           <none>
kube-system    kube-scheduler-node1                        1/1     Running   2 (6h57m ago)   7h16m   172.16.25.140   node1   <none>           <none>

Please execute sudo nft list ruleset and sudo iptables-save -t filter so I can check if there are any existing nft tables rules on the host that might be incompatible with iptables-nft tables.

ref:

egressgateway/pkg/iptables/table.go

Line 60 in 8616c7c

nftErrorRegexp = regexp.MustCompile(`^# Table .* is incompatible, use 'nft' tool.`)

@psavva
Copy link
Author

psavva commented Feb 18, 2025

rulelist.txt
iptables-save-filter.txt

@lou-lan Please find the files attached.

I think this is happening only on the Master Nodes, and seems worker nodes do not suffer from this

@lou-lan
Copy link
Collaborator

lou-lan commented Feb 19, 2025

rulelist.txt iptables-save-filter.txt

@lou-lan Please find the files attached.

I think this is happening only on the Master Nodes, and seems worker nodes do not suffer from this

In the iptables filter rule you provided, I did not see any text like ^# Table .* is incompatible, use 'nft' tool. regarding the filter table. I suspect that the incompatible nft rules no longer exist, so it may be possible to try reinstalling the egress gateway again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lou-lan lou-lan

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@psavva @lou-lan