Should Gateway API move in-tree (like Ingress)?

[shaneutt]

Gateway API as a project was a pioneer in the "creating official CRDs" movement in Kubernetes SIGs. The choice to go CRD provided velocity and autonomy and has often been touted as a good call and a boon for the project.

CRDs however have their difficulties for end-users however! So in a community meeting we revisited the question of "Should we move Gateway API in-tree?" and we took an action item to make some discussions like this.

To be more definitive this means make some subset of our APIs official APIs like Ingress is, and (importantly) make them available by default on clusters.

We hope people will please answer the poll, but also add an accompanying comment as understanding your pain points and reasons could be very illuminating for us (perhaps especially if you answer is no or I'm uncertain).

[shaneutt]

I voted YES, for a wide variety of reasons a few prominent ones being:

• I see being CRD based as a short and long term problem for adopting Gateway API
• As someone who works on a Kubernetes platform, I'm not fond of the platform dilemma of "I have to choose versions of Gateway API to ship by default" as this causes some friction with interop with Gateway API implementations running on my platform.
• It overall just becomes easier to use and document for end-users whom we've been telling for years should consider it the next generation of Ingress (note that there's also the issue that it's not as simple as Ingress for some basic cases, but I've created a separate discussion for that)

[howardjohn]

I voted "Uncertain". There are definitely benefits which others mention which I agree with.

However, there is a huge risk to velocity that I worry about. In another context I just said "X API is alpha in Kubernetes, so it will be roughly 3 years before we can use it". Our users would love to not wait 3 years for Gateway features!

Its also not clear to me what the transition looks like from CRD to core. Depending on how this works, it plausibly could solve the above problem. For instance, if you can have a CRD override the core type, you could in theory have users get the 'bundled version' (maybe older, but ok) or deploy the CRD manually and get the latest and greatest. Then you kind of have 'bundled (super stable)', 'stable', and 'experimental' I suppose. Not sure, this may cause weird fragmentation as well.

[keithmattix]

I'm also uncertain - while I'm all for making it clearer to users (especially new users) that they should be leveraging the Gateway API, I'm seriously concerned about stifling innovation. Maybe there's some solution here where GEPs are baked as CRDs and then once a resource gets through v1, it goes through a(n expedited?) KEP process. Perhaps the in-tree version is always a year behind the out of tree version or something. I'm open to discussing, but to John's point, we need to figure out precedence and the UX of merging

[shaneutt]

If our discussion reveals that the primary concern about moving in-tree is the difficulty in innovating or making progress, we may want to consider whether sidestepping these challenges will cause us (and importantly, our end-users) far more difficulties down the road. Gateway API as a project paved the way for "official CRDs" and if we pave the way back into core we may have an opportunity to help with the velocity problems therein.

[tao12345666333]

+1

I think the Gateway API will eventually become an in-tree API, but it may not be appropriate at this stage.

As an independent project, Gateway API can have its own release and compatibility policies, but if it becomes in-tree, it needs to follow the rules of Kubernetes.

Although the Gateway API has been GA for a while, I feel that it still needs to be developed and adjusted, and we still have some problems that we have not solved. For example, timeout settings. Since this project only defines the specification, if we want to make it an in-tree API, we need to make it more general and easier to implement or integrate.

[shaneutt]

Good thoughts Jintao. I agree that we have some table-stakes features missing (rate limiting, timeouts, and auth come to mind).

[aojea]

It is important to consider in this context that for any API in-tree we are requiring reference implementations in tree, because users expect the APIs to work without installing anything
NetworkPolicy and Ingress are good examples of why in-tree APIs without implementations does not work well,

[shaneutt]

Yes, we would have to think hard about this because we've had a verbal contract with the community for some time that we would not have a "default implementation". We've been open to the possibility of a "reference implementation" (see blixt) but any implementation in upstream which doesn't protect itself from production usage would potentially harm that. To expand upon what that verbal contract is, we explicitly chose to NOT have an upstream/default implementation because we expected that this would create a "level playing field" for implementations, and 5 years later it seems like it accomplished that as we have almost 3 dozen implementations and one of the most collaborative projects in Kubernetes history. We need to be very thoughtful and deliberate about the line between "reference implementation" and "default implementation" if there ends up being a lot of energy behind going down this path. Thanks for bringing this up Antonio!

[kflynn]

A metaquestion: are we really talking about "in-tree"? or are we talking about "automagically available in-cluster"?

To me, one of these is a question about how development is done, and the other is a question about what Gateway API's user experience is like. I care more about UX than I do about where exactly the development happens (though I care about that too, somewhat).

[shaneutt]

Great point, and it plays well with some of the energy around improving the CRD experience as a whole. I feel like we could have an entire discussion on just this topic (and I would like to).

[mikemorris]

Improving the CRD experience feels like something that could be very well-suited for a contributor summit unconference topic in a few weeks kubernetes/community#7993

[arkodg]

Reframing this another way - From an end user perspective, having the Ingress resource available in the cluster and the Gateway API not available (either needing to be installed by the end user, implementation or cluster/cloud provider) , in a way continues to endorse Ingress, so most users are not motivated to migrate, even if tools like ingress2gateway exist

[mlavacca]

Reframing this another way - From an end user perspective, having the Ingress resource available in the cluster and the Gateway API not available (either needing to be installed by the end user, implementation or cluster/cloud provider) , in a way continues to endorse Ingress, so most users are not motivated to migrate, even if tools like ingress2gateway exist

This is a very good point I 100% agree on

[youngnick]

I am a very hard no on moving the code to in-tree, for a few reasons.

Firstly, to explain, here I am using "in-tree" to mean the following:

• the API definitions are included in the base install of the Kubernetes API, and are no longer defined as CRDs
• the APIs are updated and released only as part of an upstream Kubernetes release (which would be necessary if they were included as in the first point)
• Changes to the APIs require the full KEP process, with production readiness review, a 12-month soak period before moving to default-enabled, and so on.
• Enabling experimental APIs requires access to the apiserver feature flags, which basically no hosted Kubernetes provider will give you (for good reason!)

Secondly, I think it's really, really important to remember that Kubernetes is, by and large, stable software. Users of Kubernetes do not want a greater velocity of change. The LTS working group has got a lot of feedback from big institutions like banks that they want slower releases, and for releases to be supported for longer. I think that speeding up of the upstream Kubernetes release cycle will not happen any time in the next 5 years, if ever. People run platforms on Kubernetes that people's lives depend on, if anything I think we're not being careful enough.

All of the above would mean that Gateway API releases will be on the same cadence as Kubernetes releases (so, three times a year), and even experimental changes will need the full KEP process - and because the things will be included, we'll need signoff from groups like SIG-API Machinery and SIG-Architecture on many changes (Policy Attachment, particularly, would need a KEP all its own, and we would need a lot more certainty around how things work before we could even get it to alpha.)

So, I'm very much a no here, because the net effect would be:

• New features would take 12-24 months to become available (with 24 months being more common), because most people can't enable feature gates in their clusters.
• The number of people required for approvals will be larger, so the approval process will be slower than it is today

There's some additional background that may not be clear to folks who haven't been following the upstream closely as well: there has been a multi-year effort to move things out of tree, to make the core Kubernetes code base smaller, so that less changes will be necessary, so that folks who want less change per version can continue using Kubernetes. That's why cloud provider code, for example, is now out of tree.

Gateway API is actually well ahead of the curve here, and the fact that we are finding the issues with distributing CRDs as an extension mechanism is going to be good for Kubernetes as a whole because we can find solutions as well, at a much faster cadence than would be possible in upstream.

With all of that said, I one hundred percent agree that we need ways to ensure that Gateway API CRDs are:

• easier to install (or ideally, come installed when you set up a cluster)
• easier to manage versions (a lot of the problem with installing comes from the possibility of having multiple implementations with multiple required versions, and that is currently very difficult to solve automatically)
• easier to include in application definitions (for example, better Helm chart support for installing HTTPRoutes would really help with adoption too!)

I just feel very strongly that moving away from delivering this API with CRDs is not worth the additional pain it would cost everyone, both us and users. Even with all their problems, using CRDs is the best option we have available, and I don't think we can build a better option in any reasonable amount of time and/or resources.

[shaneutt]

I appreciate (and I imagine readers will appreciate) your long and well thought out explanation of your position on this. To add some extra flavor for readers: Nick works on the LTS (thank you for this much needed service 🙇), and there can be no mistake that changes in Kubernetes core are especially sensitive as the project is now used in a bazillion places and can have incredible impacts on huge numbers of people.

I personally think it's a bit unfortunate that we've been increasingly expressing our need to stabilize and harden "core" by pushing new innovations down the CRD path without more deliberate paved path there (see also this related thought), but at the same time I recognize what you are saying and after several replies to this discussion (and the conversation in the community sync where this discussion broke out from) it seems increasingly obvious that "improving the CRD situation" has very little motivational contention (though we can certainly deliberate on how we improve it) so that very well may be where we need to focus the energies and motivations around this.

[robscott]

I absolutely agree with the problem statement - installing and managing Gateway API CRDs is annoying and creates friction that likely limits the adoption of Gateway API. With that said, I strongly disagree with moving this API into tree.

My rationale for this can be split into several points:

1. It's not sustainable for the Kubernetes core to indefinitely grow in size
Ultimately, to ensure that Kubernetes is sustainable and reliable, we need to aim for a stable and relatively small core. Many efforts are already underway to move projects outside of core with this in mind. That likely also translates to a slow(er) change of pace for the core upstream project moving forward.

2. The Kubernetes upstream development cycle is at least 2x slower than Gateway API
Even if we were to take the current pace of upstream development, it would take a a minimum of 4 releases (16 months) for a new feature to graduate to GA, and likely an additional 12-16 months for that Kubernetes version to be broadly used. On average, I'd expect the time between a new feature proposal and it being broadly available to be 2-3 years. In contrast, when a Gateway API version is released, it can be immediately installed on any Kubernetes cluster running one of the 5 latest Kubernetes versions (often even a larger version range than that).

3. The Kubernetes upstream development cycle does not enable the same level of collaboration
In general, every proposal needs to fit into a KEP, and every API change needs to fit into a single PR. It's very difficult to collaborate like this. When we've attempted to collaborate on API changes in the past, it's involved a few people sharing access to a fork of Kubernetes, but that simply doesn't scale.

4. It's very confusing when an API is included in Kubernetes but doesn't actually do anything by default
As we've seen with both Ingress and Network Policy, APIs that are available in upstream without a default implementation lead to a great deal of confusion. If we were to also include a reference implementation by default as well, that would indeed provide a better UX, but at the cost of going against what we've said previously in this project.

5. This change would likely result in less contributor activity in the project
There are already a fairly small set of contributors that are willing to wait for the ~year long process to get a new feature to GA in Gateway API. If that process started to get closer to 3 years, the number of contributors with that level of patience would grow closer to 0. Instead, we'd likely see the same problems we saw with Ingress - a tiny and effectively frozen API that led to tons of annotations and alternatives instead of true portability.

---

With all of that said, there are very real pain points when working with CRDs for contributors, users, and implementers of this project. I do want to find a way to dramatically improve this experience. That will undoubtedly require some work in this project to make the installation process better, and likely also some work in upstream to smooth off the rough edges of CRDs.

Given that we have a finite amount of time and people that are able to work on this project, I'd rather focus our efforts on improving the overall experience than making a significant project to upstream. For example, using CRDs does not prevent cluster providers from managing the CRDs as a cluster addon. We could work as a project to define some guidelines for how to do that safely and likely resolve much of the overall UX problems we're facing with a significantly smaller amount of work.

If in the future, there comes a point in time where this API is not meaningfully changing and has fully stabilized, we can certainly revisit this conversation, but until then, I'd strongly prefer continuing to iterate on this project with CRDs.

[shaneutt]

2. The Kubernetes upstream development cycle is at least 2x slower than Gateway API Even if we were to take the current pace of upstream development, it would take a a minimum of 4 releases (16 months) for a new feature to graduate to GA, and likely an additional 12-16 months for that Kubernetes version to be broadly used. On average, I'd expect the time between a new feature proposal and it being broadly available to be 2-3 years. In contrast, when a Gateway API version is released, it can be immediately installed on any Kubernetes cluster running one of the 5 latest Kubernetes versions (often even a larger version range than that).
3. The Kubernetes upstream development cycle does not enable the same level of collaboration In general, every proposal needs to fit into a KEP, and every API change needs to fit into a single PR. It's very difficult to collaborate like this. When we've attempted to collaborate on API changes in the past, it's involved a few people sharing access to a fork of Kubernetes, but that simply doesn't scale.
5. This change would likely result in less contributor activity in the project There are already a fairly small set of contributors that are willing to wait for the ~year long process to get a new feature to GA in Gateway API. If that process started to get closer to 3 years, the number of contributors with that level of patience would grow closer to 0. Instead, we'd likely see the same problems we saw with Ingress - a tiny and effectively frozen API that led to tons of annotations and alternatives instead of true portability.

I understand the perspective, but I wish we were all more motivated to help change this. See more of my thoughts here and here.

4. It's very confusing when an API is included in Kubernetes but doesn't actually do anything by default As we've seen with both Ingress and Network Policy, APIs that are available in upstream without a default implementation lead to a great deal of confusion. If we were to also include a reference implementation by default as well, that would indeed provide a better UX, but at the cost of going against what we've said previously in this project.

No argument. And for anyone reading this, see Antonio's somewhat related post.

With all of that said, there are very real pain points when working with CRDs for contributors, users, and implementers of this project. I do want to find a way to dramatically improve this experience. That will undoubtedly require some work in this project to make the installation process better, and likely also some work in upstream to smooth off the rough edges of CRDs.

Given that we have a finite amount of time and people that are able to work on this project, I'd rather focus our efforts on improving the overall experience than making a significant project to upstream. For example, using CRDs does not prevent cluster providers from managing the CRDs as a cluster addon. We could work as a project to define some guidelines for how to do that safely and likely resolve much of the overall UX problems we're facing with a significantly smaller amount of work.

If in the future, there comes a point in time where this API is not meaningfully changing and has fully stabilized, we can certainly revisit this conversation, but until then, I'd strongly prefer continuing to iterate on this project with CRDs.

You're long and well thought out response is deeply appreciated, and I expect will be very helpful to readers finding there way here because they're asking themselves the question "why is this not in-tree?" to better understand the perspectives on it from people who work on it day-to-day.

You said it, and as I review this discussion and the original meeting discussion where it came up it seems increasingly obvious to me that there's not very much contention around the "CRDs need help" problem statement. So that's for continuing to help highlight that, and maybe it's there where more of our attention belongs 🤔

[mlavacca]

Even though I totally agree with the root problem and the statement of making life easier for our users and increasing the Gateway API adoption, I'm opposed to move Gateway API in-tree.

I do share the concerns about the development velocity and the fact that moving Gateway API upstream would significantly lower our process, with some bad side effects, such as the potential loss of contributors, and the rise of implementation-based alternatives (in a nutshell what already happened to ingress API).

I think we should spend our energy on improving the CRD management both upstream and in the Gateway API. One point is shipping Gateway API CRDs with core Kubernetes. I think it'd heavily help increase adoption while keeping Gateway API independent from upstream cycles, needs, and rules. A user who creates a new cluster will have Gateway API CRD at hand to play with, and in case they want to upgrade to the newest Gateway API version (or change channel) they'll be able to do it as they always did.

[shaneutt]

Your thoughts and perspective are greatly appreciated. Github discussions may not have an good alternative for me, but since I have two responses to other posts that I think are relevant to this one I'll just link one and the other here.

[skitt]

I agree with Rob’s points, and for those reasons I don’t think moving Gateway API in-tree would be a good idea in the short to medium term.

I also think that Gateway API is a strong enough community with a large base of interested users for Kubernetes to use it to fix the issues around CRD usage! (Note I’m not saying the Gateway API community needs to fix those, but that the Kubernetes community needs to, and that Gateway API can hopefully be a good enough reason to find the energy to make the appropriate changes once we figure out what they are.)

[shaneutt]

I also think that Gateway API is a strong enough community with a large base of interested users for Kubernetes to use it to fix the issues around CRD usage!

This part resonates a lot with me, and I have a lot of appetite for trying to personal help make this better. It also seems like the most common thread among responses to this discussion.