Kong Ingress Controller cannot sync dataplane once inexistent plugin is introduced into kubernetes

[dlamotte]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

When a plugin that is not valid in running kong gateway's is referenced, it does not impact critical data plane updates to kong gateways (including pod IPs in services/endpoints updates; upon becoming stale result in odd errors including timeouts or responses from incorrect pods that happen to also speak the protocol [like http]).

2024-09-17T18:36:15Z	info	controllers.HTTPRoute	HTTPRoute has been configured on the data-plane	{"GatewayV1HTTPRoute": {"name":"mock","namespace":"kong-ingress-controller"}, "v": 0, "namespace": "kong-ingress-controller", "name": "mock"}
2024-09-17T18:36:16Z	error	Failed to fill in defaults for plugin	{"url": "https://10.244.0.16:8444/", "plugin_name": "not-a-plugin", "error": "error retrieveing schema for plugin not-a-plugin: HTTP status 404 (message: \"No plugin named 'not-a-plugin'\")"}
2024-09-17T18:36:17Z	error	recording a Warning event for object	{"name": "not-a-plugin", "namespace": "kong-ingress-controller", "kind": "KongPlugin", "apiVersion": "configuration.konghq.com/v1", "reason": "KongConfigurationApplyFailed", "message": "invalid name: plugin 'not-a-plugin' not enabled; add it to the 'plugins' configuration property", "error": "object failed to apply"}
2024-09-17T18:36:17Z	error	dataplane-synchronizer	Could not update kong admin	{"error": "performing update for https://10.244.0.16:8444/ failed: failed posting new config to /config: got status code 400"}

This affects data plane availability of services to allow any namespace to taint the gateways in a way to break critical update functionality.

Expected Behavior

Kong gateways continue to serve traffic as if the bad plugin is not configured (minimal blast radius; the specific http route referencing the bad plugin is not updated and the previous version of it is retained).

Steps To Reproduce

Apply a `KongPlugin` and `HTTPRoute` which references it. Critically, that `KongPlugin` cannot be a valid plugin that the Kong Gateway knows about.

Practically, this occurs when kubernetes templates are applied prior to the plugin being available or if plugins are being removed from the gateway and references still exist.

Kong Ingress Controller version

v3.1.4

Kubernetes version

Client Version: v1.29.0
Server Version: v1.29.2

Anything else?

No response

[dlamotte]

Possibly related:

• Upstream IP's not updated #773
• After updating deployment, controller failed to sync pod to kong #1025

[randmonkey]

@dlamotte If a KongPlugin is using a non-exist plugin, it will be rejected by KIC's webhook. So please confirm that how did the KongPlugin be created in your cluster.
Also, we introduced FallbackConfiguration feature to implement the requirement to just remove the HTTPRoute affected. This requires enabling FallbackConfiguration feature gate.