RuntimeConfig one-pager

This one-pager proposes a lightweight implementation of the PRI
proposal, along with an alternative for ControllerConfig.

I believe this will be relevant for the in-flight Composition Functions
beta design, as well as for Providers.

crossplane#3601
crossplane#2671
crossplane#4306

Signed-off-by: Nic Cope <[email protected]>

design/one-pager-package-runtime-config.md

@@ -0,0 +1,222 @@
+# Package Runtime Config
+
+* Owner: Nic Cope (@negz)
+* Reviewers: TODO
+* Status: Accepted
+
+## Background
+
+Crossplane Providers are Kubernetes controller managers. They're packaged in an
+OCI container and expect to connect to an API server. You install a Provider
+declaratively by creating a Provider resource. When you do, the Crossplane
+package manager installs your provider by creating a Kubernetes Deployment,
+along with some accoutrements like a ServiceAccount and a Service. The RBAC
+manager also creates some RBAC ClusterRoles and ClusterRoleBindings to allow the
+Provider's ServiceAccount (and thus Deployment) to do what it needs to do.
+
+There are two things we'd like to improve about the way this works today:
+
+1. Folks want more control over how their Providers are deployed - over the
+   configuration of the Deployment (etc) Crossplane creates. ([#3601])
+2. In some cases, folks don't want Crossplane to create a Kubernetes Deployment
+   at all. They want to run a Provider some other way. ([#2671])
+
+Soon [Composition Functions][functions-beta-design] will also be deployed by the
+package manager as Kubernetes Deployments. We expect they'll have similar
+requirements. Since Functions aren't Kubernetes controllers, in this document
+I'll refer to the long-running processes some packages need to deploy as
+'package runtimes'.
+
+Crossplane has a v1alpha1 ControllerConfig type that addresses the first issue
+for Providers. It has been marked deprecated, to be removed if and when we find
+a suitable replacement. We deprecated ControllerConfig because:
+
+* It was growing piecemeal to support templatizing an entire Deployment.
+* We think in some rare cases package runtimes won't use Deployments.
+
+It's worth noting that the desire to deploy a package runtime as anything other
+than a Kubernetes Deployment in the same cluster where Crossplane is running is
+_quite rare_. To my knowledge only Upbound currently has this requirement.
+
+## Goals
+
+The goals of this design are to:
+
+* Give folks full control over package runtime Deployments.
+* Make it possible to run package runtimes as something other than a Deployment.
+
+## Proposal
+
+I propose a new flag to Crossplane: `--package-runtime`. This flag would have
+two possible values (at least to begin with):
+
+* `--package-runtime=Deployment` (default) - Create a Kubernetes deployment.
+* `--package-runtime=Disabled` - Do nothing.
+
+When running in Deployment mode the package manager will function as it does
+today. It will create a Kubernetes Deployment for each package runtime, plus any
+additional supporting configuration such as a ServiceAccount, etc.
+
+When running in Disabled mode the package manager won't create a package runtime
+at all. It will create a revision (e.g. a ProviderRevision) and deliver the
+package's payload (e.g. a Provider's CustomResourceDefinitions), but do nothing
+else. In Disabled mode it's expected that an external controller will take care
+of reconciling the relevant package revision to deploy a package runtime however
+it sees fit.
+
+I also propose we replace ControllerConfig with a new DeploymentRuntimeConfig
+type in the pkg.crossplane.io API group. This type is used to configure the
+package runtime when the package manager is running with
+`--package-runtime=Deployment`.
+
+A DeploymentRuntimeConfig is referenced from any package that uses a runtime,
+such as a Provider. For example:
+
+```yaml
+apiVersion: pkg.crossplane.io/v1
+kind: Provider
+metadata:
+  name: provider-example
+spec:
+  package: xpkg.upbound.io/crossplane-contrib/provider-example:v1.0.0
+  runtimeConfigRef:
+    apiVersion: pkg.crossplane.io/v1
+    kind: DeploymentRuntimeConfig
+    name: default
+```
+
+There's a few things to note here:
+
+* The referencing field is `spec.runtimeConfigRef`. A DeploymentRuntimeConfig is
+  one possible type of runtime config - for now, the only supported one.
+* Because there could in future be other kinds of runtime config the reference
+  requires an `apiVersion` and `kind`.
+
+If the `runtimeConfigRef` is omitted it will default to a runtime config named
+"default" of the type specified by the `--package-runtime` flag. For example
+when Crossplane is run with `--package-runtime=Deployment` a Provider will use a
+DeploymentRuntimeConfig. This behavior matches that of ProviderConfigs. If you
+omit the `providerConfigRef` when creating an MR Crossplane defaults to using a
+ProviderConfig named default. 
+
+Automatically setting a default runtime config has two advantages:
+
+* The default configuration is no longer [hardcoded][hardcoded-pkg-deployment]
+  into the package manager.
+* Administrators can override the configuration all runtimes use by default.
+
+The Crossplane init container will create a default DeploymentRuntimeConfig at
+install time. A Crossplane administrator could then replace it with their own.
+For example Crossplane might install a default DeploymentRuntimeConfig that
+limits all package runtimes to 1 CPU core, but a Crossplane administrator might
+wish to change this to give all runtimes 2 CPU cores by default. Individual
+packages can still be explicitly configured to use a specific runtime config.
+
+Given that we saw ControllerConfig growing into a template for a Deployment, I
+propose we lean into that and make DeploymentRuntimeConfig exactly that. For
+example:
+
+```yaml
+apiVersion: pkg.crossplane.io/v1alpha1
+kind: DeploymentRuntimeConfig
+metadata:
+  name: default
+spec:
+  deploymentTemplate:
+    metadata:
+      labels:
+        example: label
+    spec:
+      replicas: 1
+      template:
+        securityContext:
+          runAsNonRoot: true
+          runAsUser: 2000
+          runAsGroup: 2000
+        containers:
+          # The container used to run the Provider or Function must be named
+          # 'package-runtime'. The package manager will overlay the package's
+          # runtime image, pull policy, etc into this container.
+        - name: package-runtime
+          securityContext:
+            runAsNonRoot: true
+            runAsUser: 2000
+            runAsGroup: 2000
+            privileged: false
+            allowPrivilegeEscalation: false
+          ports:
+          - name: metrics
+            containerPort: 8080
+    # A DeploymentRuntimeConfig can also be used to configure the Service and
+    # ServiceAccount the package manager creates to support the Deployment.
+    serviceTemplate: {}
+    serviceAccountTemplate: {}
+```
+
+The above DeploymentRuntimeConfig matches the values that are currently
+[hardcoded into the package manager][hardcoded-pkg-deployment]. The
+`deploymentTemplate`, etc fields are similar to a Deployment's `template` field.
+
+The package manager will always be opinionated about some things, and will
+overlay the following settings over the top of the provided template:
+
+* The name and namespace of the deployment, service, and service account.
+* The image, image pull policy, and image pull secrets (set from the package).
+* The label selectors required to make sure the Deployment and Service match.
+* Any volumes, env vars, and ports required by a runtime (e.g. for webhooks).
+
+## Future Improvements
+
+The `--package-runtime` flag and `runtimeConfig` API are intentionally designed
+such that other runtimes _could_ be added to Crossplane either natively, or as
+PRI plugins in future (See [Alternatives Considered](#alternatives-considered)).
+This allows us to prototype new runtimes implemented by controllers running
+_alongside_ Crossplane before potentially moving them into tree later.
+
+Assume for example there was a desire to use [Google Cloud Run][cloud-run] as a
+package runtime: to deploy Providers to Google Cloud Run rather than to the
+Kubernetes cluster where Crossplane is running. Under this design someone could
+write a controller, deployed alongside Crossplane, that reconciled a
+ProviderRevision by running its controller OCI container in Cloud Run. To do so
+they would just need to set `--package-runtime=External` to let Crossplane know
+ProviderRevisions were handled by an external system.
+
+The 'external' cloud run package runtime controller could introduce its own
+CloudRunRuntimeConfig custom resource that Providers could use to configure how
+they should be deployed to Cloud Run:
+
+```yaml
+apiVersion: pkg.crossplane.io/v1
+kind: Provider
+metadata:
+  name: provider-example
+spec:
+  package: xpkg.upbound.io/crossplane-contrib/provider-example:v1.0.0
+  runtimeConfigRef:
+    apiVersion: pkg.example.org/v1
+    kind: CloudRunRuntimeConfig
+    name: default
+```
+
+If there was sufficient demand, the functionality of the external cloud run
+controller could be later built-in as `--package-runtime=GoogleCloudRun`.
+
+## Alternatives Considered
+
+The primary alternative to this proposal is the Provider Runtime Interface RFC
+captured in [#2671]. This RFC intends to make it possible to use other package
+runtimes besides a typical Kubernetes deployment, but implies Crossplane would
+be responsible for deploying such runtimes via an abstraction layer.
+
+I believe adding the `--package-runtime` flag achieves the spirit of this
+proposal without introducing any additional complexity or indirection into
+Crossplane. Given how niche the desire to use alternative runtimes is I feel
+it's reasonable to expect anyone who wants one to implement it using their own
+controller.
+
+
+[#3601]: https://github.com/crossplane/crossplane/issues/3601
+[#2671]: https://github.com/crossplane/crossplane/issues/2671
+[functions-beta-design]: https://github.com/crossplane/crossplane/pull/4306
+[hardcoded-pkg-deployment]: https://github.com/crossplane/crossplane/blob/v1.12.2/internal/controller/pkg/revision/deployment.go#L60
+[google-cloud-run]: https://cloud.google.com/run