Improving Discoverability of Policy Attachment

[robscott]

The Gateway API policy attachment model is the primary extension mechanism of the API. Unfortunately we have not yet provided a good way for users to discover policies that are available or applicable to their Gateway API resources or a good way for implementations to communicate that. Starting a discussion to gather ideas for how we can improve this. I'll start it off with a few ideas we've already talked about, but please don't hesitate to add your own ideas here.

[robscott]

We could introduce a PolicyBinding resource that every implementation of Policy Attachment would be responsible for publishing. Every time a policy attachment was implemented, the implementation would also publish a PolicyBinding resource that described the following:

1. The kind, name, and namespace of the target
2. The kind, name, and namespace of the policy
3. The type of policy (direct or hierarchical)
4. The controller that implemented the attachment
5. The status of that attachment

This would enable anyone to run kubectl get policybindings and see all Gateway API policies that have been applied to any resource in their cluster.

[keithmattix]

I'm hesitant to have the binding be per policy instance instead of type; feels like a lot of overhead. Instead, couldn't we have a PolicyRegistration for each type of policy (i.e. CRD) for discovering the kinds of policy that exist? From there, other tooling (such as the kubectl plugin proposed below) could list all the policy registrations to compute the runtime configuration on a per-resource level

[sunjayBhatia]

This is somewhat similar to the last general idea here: #1531 (a resource provided by Gateway API that represents the attachment of a policy, not the actual implementation-specific values, I like the naming you have, calling it a PolicyBinding)

[robscott]

@keithmattix I think that's also similar to the idea I added below for each GatewayClass to populate in status which resources it supports. Admittedly we're still lacking a comparable resource for mesh, so we'd still need to solve for that.

I think we're roughly circulating around the following similar but different ideas:

1. PolicyBinding resource for every instance of policy attachment that is implemented by a controller.
2. PolicyRegistration resource (or GWC status) that describes the policies that exist. Note this was supposed to be covered by predefined labels on every policy CRD (implemented by https://github.com/michaelvl/gateway-api-lens), so maybe an alternative is making that more discoverable and visible (ie kubectl plugin).
3. EffectivePolicyConfigurationReferences - PolicyBinding that also includes the computed policy associated with each instance of policy attachment.

I agree that both 1 and 3 would add significant overhead, but I think the scope of this would be limited to 1:1 mapping of each instance of a policy, so theoretically not massive in terms of scalability.

[mikemorris]

Cross-referencing @LiorLieberman's proposal #2108 here - having an implementation advertise known policy types feels like a much nicer UX than adding more CRDs on top of an already-complex extension mechanism.

[robscott]

We could put more effort into tools to visualize the full Gateway API resource graph present in a cluster, including attached policies. For example, https://github.com/michaelvl/gateway-api-lens provides a great example of this kind of approach.

[robscott]

We could develop a kubectl plugin that could do at least some of the following:

1. Describe all policies that apply to a single resource (including hierarchical policies attached above and below)
2. Describe the effective policy that applies to a single resource by computing the result of hierarchical policies and merging with direct policies
3. List all types of policies that are available (potentially per-GatewayClass)
4. Other ideas welcome

[bowei]

This one seems like super low barrier to entry for contributors who want to try out what information they would find useful before we do something more costly like adding new APIs.

[keithmattix]

I'm inclined to agree; this kind of discovery work seems low effort, high impact for helping us to know what should be included in a standard API

[vtrenton]

The thing I'm not a fan of with this approach is it puts discovery/observability on the client side. As silly as it seems this implies people are using kubectl to manage clusters. Some people might be using custom tooling or something like curl. For this - I personally think using gateway API status fields makes the most sense. Third party tooling can be built around aggregating and visualizing this data (yes this even would possibly include a kubectl plugin). I feel it more important we provide this data in a tooling agnostic manner.

[bowei]

CLI is an easy way to experiment -- APIs are forever once we add something. (Yes, even "experimental" has a cost)

[robscott]

We could introduce a standard section to status of all Gateway API resources that shows which policies have been applied to the resource or section of that resource. At least for Routes, this would need to be namespaced by at least controller name/parentRef in the same way the rest of Route status is.

[sunjayBhatia]

Related: #1531

One missing option from that discussion is just listing the names of the applied Policies on a resource

[youngnick]

We could do this with a Condition as well; that has the advantage that we could use a namespaced version on core objects as well (so, PolicyAffected or something for Gateway API objects, and gateway.networking.kubernetes.io/PolicyAffected or <controllername>/PolicyAffected or similar for core objects like Secret, which all have Conditions now.) This would be effectively a signal saying "One or more policies are affecting this object".

[shaneutt]

If we go forward with what we have, this one in particular seems very axiomatic, an obvious thing to do.

[mikemorris]

The difficulty with this approach is if PolicyAttachment is used to target other resources, such as Service - and ending up with two different patterns for discoverability.

[robscott]

We could introduce a new element in GatewayClass status that could be used to publish the kinds of policies supported by a GatewayClass.

[sunjayBhatia]

Related to #2012 (comment) and #2012 (comment) would be a new field on GatewayClass/Gateway/*Route resources that would allow the resource owner to specify what policies are allowed to be bound to a particular resource, similar in concept to Gateway Listener allowedRoutes

This could be combined with the linked ideas and may alleviate some of the concerns from #2015

[sunjayBhatia]

Already linked to the above comments, but shameless plug for my earlier discussion which relates to this one: #1531

Particularly I'm fond of proposing the idea of adding EffectivePolicyConfigurationReferences (name tbd of course) to Gateway API resource status, which would be a reference to a resource with the actual values for a particular Policy kind that are in effect on a target resource (e.g. an implementation would create an instance of a FooPolicy or create a new FooPolicyState whose values actually represent coalescing all of the hierarchical/direct FooPolicies applied to a HTTPRoute). Since these are implementation-specific CRDs/values/etc., it is much easier to have actual state (with the desired state in Policies targeting this resource) represented this way in a generic object reference vs. cramming it into Status on a generic resource.

[sunjayBhatia]

This idea as-is doesn't really work with applying Policies to non-Gateway API resources (e.g. Service) so that is a detail to work out tbh

[robscott]

I actually think status may be a reasonable place to have a map[string]string that enables each controller to populate the effective policies it is applying. Agree that this doesn't really translate to Service, but this is something that could theoretically be added later as it's more of a nice to have than a requirement for resources that policies could be attached to.

[shaneutt]

Take a look at the PR @kflynn recently posted which is relevant:

#2015

I know this topic is going to be at least a little controversial because it suggests a significant pivot, but I personally find this juxtaposition very persuasive and compelling: what if we can solve the problem by partially not having the problem? I would definitely encourage people to chime in there as I think it's a very worthwhile discussion.

[michaelvl]

One of the strengths of policy attachments is that is supports the multi-role nature of the gateway api well. Changing the approach to have explicit references in resources (the 'pivot') probably makes this more difficult. E.g. how do an infrastructure operator apply a policy to a user-defined gateway resource if the policy cannot reference the gateway but instead the gateway need to reference all policies that applies to it. It may be possible to come up with a good model, and IMHO its an important characteristic to consider.

[youngnick]

A couple of other ideas I thought of recently:

We could mandate that implementations that support any Policy objects have a Validating Admission Webhook that will return a message (warning maybe?) when an applied object is affected by some Policy, which may be an inherited or indirect one.

Pro:

• This gives object owners a very clear signal that something some Policy is going to affect their object, at apply time, which helps a lot with discoverability. It could conceivably say which Policies, even.

Cons:

• Implementations would have to have a webhook, which is another thing to run
• The webhook will need to have the same data model that the implementation uses, and keep track of which GatewayClasses, Gateways, Routes, and Policies are relevant.

This is less of a good idea (too many cons), but for completeness: We could also suggest that, for Policies that affect fields inside an object, they include a Mutating Admission controller that would update the applied object with the settings generated by the resultant set of policy.

Pro:

• The object you see or retrieve from the apiserver is the settings that the object is producing. This makes it pretty clear what you're actually getting from your object, since the exact settings that are in there are the actual ones.

Cons:

• Once the object is modified once, there's no way for Kubernetes (or any controllers) to know if the setting was originally set by the user or by some mutating controller. So if, say, the mutating controller sets a default, which gets written into the object, there's no way subsequently to tell if that value was from the default or the user. And if the default is later updated, controllers have no way to know which values should be changed.
• Mutating also runs the risk of having different controllers "fighting" over some value, and changing it back and forth.

[sanjaypujare]

...an inherited or indirect one...

I understand the inherited policy but what's the "indirect" one? Other than inheritance how can a policy indirectly affect an object to which it is not attached?

[youngnick]

Given that the other type of policy is Direct, I've heard some folks using "Indirect" instead of inherited, so I was trying to make it clear that the two are synonyms. I think it's ended up more confusing though, sorry.

[brianehlert]

With the Policy that we have had in our Ingress CRDs for the past few years we handle the attach at the equivalent of the Route level.
This is to put the Route owner in control of the attach for debugging purposes. It is also easier to pull the configuration together.
And the Policy should indeed be re-usable.

The business problem is that the Policy author needs a way to attest to their Policy being applied.

In our case we use Policy as a way to handle WAF rules, AuthN / AuthZ, mTLS, etc.
In the case of WAF rules, those could be compliance related, API enforcement related, etc.