Gateway API Plugins

[shaneutt]

Many proxies have some sort of "plugin" capability, perhaps the most well-known mechanism I am aware of being WASM plugins via envoy proxy. A "plugin" in this context is generally some bit of custom logic which can be applied to traffic (e.g. "filters" and so forth) where the logic is injected via some object (e.g. WASM) which commonly is provided by end-users to enable custom logic beyond what the proxy originally intended.

Note: use of the word "proxy" here is due my own familiarity with in-cluster implementations, however
we should be considering this as applicable to other implementations and load-balancers.

Several proxies out there already support some kind of plugin capabilities:

• https://istio.io/latest/docs/reference/config/proxy_extensions/wasm-plugin/
• https://doc.traefik.io/traefik/plugins/
• https://docs.konghq.com/gateway/latest/kong-plugins/

Plugin capabilities, as defined here are potentially viable in both ingress and mesh uses cases.

I bring this up because I am interested in providing some standards for how plugins work in Gateway API (at some level) such that end-users have a generally familiar "Kubernetes" way of doing this regardless of which implementations they use.

My current goals are:

• provide a definition of "plugin" for Gateway API at a documentation/specification level
• expand on how an HTTPRoute can have one or many "plugins" attached to it in the API, and provide rules for implementations
• make the API dynamic so that it can cover several plugin "types" (e.g. WASM, Lua, Go, e.t.c.) to be inclusive and enable future growth
• ensure that we consider both ingress and mesh use cases
• prescribe a mechanism for delivery of plugin objects (code, object code, e.t.c.)

Note: my current hope is that in conjunction with this work, we can work together throughout the greater
OSS community to enable a standard for delivery of plugin objects. I'm particularly interested in using
container images for proxy delivery, but if that's not something the community is generally in favor of I'm
flexible and can be persuaded to an alternative.

There are some potential parallels to draw with Policy Attachment, by comparison my hope is that we can provide something that is:

• an official part of Gateway API, and therefore highly structured
• ideally the above means we could have some plugins on the simpler end that are literally portable between implementations, not sure if we can make it pan out but I'm interested in thinking through it.
• I'm particularly interested in an attachment mechanism that is explicit, e.g. the resources have attachments in their specification.

Ultimately, we may be able to make a GEP out of these goals after we've had some time here to discuss it and iterate on them, and after we've generally gathered some consensus. It's not even clear to me yet if we even have enough common ground and portability to really make this happen, so that's basically the main open question of this discussion. Please do share your thoughts and feedback, and I would particularly like to hear from implementations who currently have a lot of plugin usage going on in their end-user ecosystems.

Note: I understand that we're currently trying to make a bee-line to our GA release, so I don't expect any
specific time-frame on this, and in fact expect that discussion will be a slow process over several months. That said,
I think it would be prudent for us to take some time prior to GA to consider how this might work, so that we don't
miss an opportunity prior to GA to make a critical change that would help us support plugins in future iterations.

[tao12345666333]

SGTM.

Usually, when using a Plugin, we would like to intervene at specific execution phases. This is similar to defining multiple request processing phases in NGINX.

Of course, if we don't consider the execution phase, there can also be some simple mechanisms, similar to the forward-auth implemented in Ingress-NGINX. The request is forwarded to the plugins for processing and the response is returned to the client.

[youngnick]

I think that maybe a place to start is to talk about what we're talking about when we say "plugin". We already have a number of extension points in the API, what's different here?

Is this an arbitrary piece of code that can run at any point in the request routing process?

If so, we're going to have to sit down and give the request routing process phases names (this is only kind of implied at the moment).

If not, we need to talk about what a plugin is. Is a custom Filter a plugin?

I guess I'm trying to say that I think that spending some time on basic terms here will pay off massively.

I'd also like to see us not end up with multiple ways of configuring the same thing - I'll be honest that my first intuition about most WASM use cases I can think of are custom filters rather than something more generic, but I'm happy to talk more and be proved wrong.

[sanjaypujare]

If not, we need to talk about what a plugin is. Is a custom Filter a plugin?

I agree. Instead of adding this new concept called "plugin", we should think about how custom filters offer the same functionality.

[shaneutt]

I'm open to this interpretation, as long as we end up with the most portable thing feasible. @sanjaypujare for the implementation(s) which you work on, are "plugins" relevant and something you'd like to see a portable upstream implementation of? If so, what implementations, and can you provide some more details?

[sanjaypujare]

...for the implementation(s) which you work on, are "plugins" relevant ..

Since I am not in favor of adding the notion of a "plugin", I don't know how to answer this question. I do see many use cases for custom filters through the use of ExtensionRef. Some examples:

• In GCP Gateways there is something called GCPBackendPolicy which is used to specify session affinity for certain services. However there are use-cases where a user would like to specify session affinity a bit "higher" in the config hierarchy such as a route rule. The reason is in case of traffic splitting we would like the session affinity as a property of the route rule instead of specific services so it overrides traffic splitting. e.g. in the following snippet the session affinity is applied for all traffic matching the path /part1/part2/resource3 regardless of where the initial request was sent (svc1 or svc2) as part of traffic splitting. This is achieved through the custom filter GCPSessionAffinityFilter.

  rules:
  - matches:
    - path:
        type: Exact
        value: /part1/part2/resource3
    filters:
    - type: ExtensionRef
      extensionRef:
        group: networking.gke.io
        kind: GCPSessionAffinityFilter
        name: ssa-filter1
    backendRefs:
    - name: svc1
      port: 50051
      weight: 90
    - name: svc2
      port: 50051
      weight: 10

• something similar for applying various security or other policies (e.g. rate limit). But instead of applying them to the whole HTTPRoute or just a service a user would like to apply them only for a specific route rule. e.g. the following snippet is structurally very similar to the one above.

  rules:
  - matches:
    - path:
        type: Exact
        value: /part1/part2/resource3
    filters:
    - type: ExtensionRef
      extensionRef:
        group: networking.gke.io
        kind: OAuthFilter
        name: oauth-filter1
    backendRefs:
    - name: svc1
      port: 50051
      weight: 90
    - name: svc2
      port: 50051
      weight: 10

In the above cases, the implementation (i.e. GCP gateways) is providing these custom filters which a user can just use and provide some config values e.g.

apiVersion: networking.gke.io/v1
kind: GCPSessionAffinityFilter
metadata:
  name: ssa-filter1
spec:
  statefulGeneratedCookie:
    cookieTtlSeconds: 50

So that's the use-case for custom filters.

By a "plugin" do you mean an extension where an end-user can provide the code as well (as suggested in #2275 (comment)) ?

[shaneutt]

By a "plugin" do you mean an extension where an end-user can provide the code as well

That is generally what I'm getting at, yes.

[sanjaypujare]

where an end-user can provide the code as well
That is generally what I'm getting at, yes.

I don't see a use case for that (yet).

[arkodg]

thanks for raising this, great points made by everyone
here's how I view this
Plugin = Intent via an Extension + Artifact to Load

so from a Gateway API perspective, it would be great to give guidance to implementations on a standardized way to download artifacts, something like

artifactRef:
  type: "application/vnd.wasm.config.v1+json"
  url: "oci://docker.io/foo/my-wasm"

similar to the guidance given for PolicyAttachment

[spacewander]

IMHO, we can divide the Plugin into three categories by how the data plane calls the Plugin:

1. Call a function in the DP. For example, the builtin Go middleware in Traefik, the C++ plugin in Envoy
2. RPC to a sidecar server. For example, the ext_proc in Envoy, the plugin server in Kong/APISIX.
3. Loading a script or binary, and run it. The script is something that can be executed by an interpreter. The binary is something that is self-contained, like a .soor a .wasm. For example, the Wasm/Go extensions in Envoy, the Lua/Wasm Plugin in Kong/APISIX, and the dynamical Go Plugin in Traefik.

For each category, we need type and config.
For the first category, we need name to indicate which function is the entry point.
For the second category, we need name to indicate which action should be executed in the sidecar server. Envoy supports configuring the RPC server address at per-plugin level. But (AFAIK) other implementations need to hardcode the server address in a static configuration. So we can't specify the RPC server address at the plugin level.
For the third category, the script/binary can be loaded from a local file system or from a remote repo. So we need a url to locate the artifact. To make it simple, we can use name as the url, so that all the plugins can have a name field.

In conclusion, we can use type, name and config to locate and configure a runnable module, which is usually called Plugin.
type is implementation-specific.
name can be a function name, a file://xxx to a plugin directory, or a oci://xxx to a remote Wasm file.
config is a KV structure that will be marshaled into the Plugin's configuration.

[keithmattix]

Thanks to @howardjohn for directing me to this discussion; I didn't know it existed.

IMO, the biggest distinction between "plugins" and "policies" is that in the former, the user injects custom data path processing AND that user wishes to control the stage within the datapath where this custom processing takes place. By contrast, policies are control-plane-managed configuration that provide user-configurable abstractions for the purposes of influencing the parameters of the policy (not the actual implementation).

One of the open questions we would probably need to tackle is figuring out a standard way to represent the ordering/stages of proxy data path processing. Each proxy has different capabilities and likely slightly different ordering,, so it would be interesting to see what the overlap is

[youngnick]

Yeah, based solely on my experiences looking at timeouts, I feel like this may be harder than it would seem to be at first glance.

[keithmattix]

Oh this is an incredibly difficult task to take on at a spec level. Not saying we shouldn't do it, but it's certainly a hard problem