Executors: add native k8s executors (#279)

* Move dind executor chart into new subdirectory * Add native executor charts * Cleanup from move * Add TODO * configures native executors * move registry into executor chart * conditionally mount docker config * move rbac into main chart * remove subdir * Fix test * yml -> yaml` * some more tests * rename dir native->k8s * set image tag * move rbac value into executor block * remove image name from tag * docs * update schema validation file * cleanup dind * gen docs * add readme with brief chart descriptions * slight change to validation script * bump rgistry disk size and add to dind values * gen docs * rip out private registry for k8s impl * rename rolebinding file\ * clarify dind in docs * change default tag of dind image * gen docs * clarify frontend url * gen docs * Specify implementation in chart names * Add separate step for executor helm chart packaging * Add separate step for executor helm chart packaging in insiders release * small doc tweak * Add changelog for executors * Fix SA name in role binding * Actually use value for dockerAddHostGateway * Set secret as env var on deployment * Use storageclass value in PVC * update docs
sourcegraph · Jun 27, 2023 · 12bb419 · 12bb419
1 parent 7cdad50
commit 12bb419
Show file tree

Hide file tree

Showing 34 changed files with 949 additions and 170 deletions.
diff --git a/.github/workflows/gcs_chart_publish_insiders.yml b/.github/workflows/gcs_chart_publish_insiders.yml
@@ -53,8 +53,11 @@ jobs:
           sed -i 's/appVersion:.*/appVersion: insiders/g' charts/*/Chart.yaml
           sed -i '/^version:/ s/"$/-insiders.${{ steps.metadata.outputs.shortSHA }}"/' charts/*/Chart.yaml
 
-      - name: Package helm charts
-        run: for i in charts/*; do helm package -u $i; done
+      - name: Package Sourcegraph helm charts
+        run: for i in charts/*; do [ -f "$i/Chart.yaml" ] && helm package -u $i; done
+
+      - name: Package Sourcegraph executor helm charts
+        run: for i in charts/sourcegraph-executor/*; do [ -f "$i/Chart.yaml" ] && helm package -u $i; done
 
       - name: Publish chart if it doesn't already exist in the repository
         run: |

diff --git a/.github/workflows/gcs_chart_publish_release.yml b/.github/workflows/gcs_chart_publish_release.yml
@@ -44,8 +44,11 @@ jobs:
       - name: Set up helm repo
         run: helm repo add sg gs://${{ env.BUCKET_NAME }}/${{ env.BUCKET_PATH }}
 
-      - name: Package helm charts
-        run: for i in charts/*; do helm package -u $i; done
+      - name: Package Sourcegraph helm charts
+        run: for i in charts/*; do [ -f "$i/Chart.yaml" ] && helm package -u $i; done
+
+      - name: Package Sourcegraph executor helm charts
+        run: for i in charts/sourcegraph-executor/*; do [ -f "$i/Chart.yaml" ] && helm package -u $i; done
 
       - name: Publish chart if it doesn't already exist in the repository
         run: |

diff --git a/charts/sourcegraph-executor/CHANGELOG.md b/charts/sourcegraph-executor/CHANGELOG.md
@@ -0,0 +1,7 @@
+# Changelog
+
+<!-- START CHANGELOG -->
+## Unreleased
+
+* Added new chart `sourcegraph-executor-k8s` to deploy Sourcegraph executors that use Kubernetes jobs.
+* **BREAKING:** Renamed `sourcegraph-executor` chart to `sourcegraph-executor-dind` to indicate these are Docker in Docker executors. To update to newer versions of this chart, ensure the new Chart name is used.
diff --git a/charts/sourcegraph-executor/README.md b/charts/sourcegraph-executor/README.md
@@ -1,102 +1,16 @@
-<!--
-  DO NOT EDIT README.md directly.
-  README.md is automatically generated from README.md.gotmpl
--->
+# Sourcegraph executor helm charts
 
-# Sourcegraph Exexutor Helm Chart
+This directory contains two Helm charts for deploying executors on Kubernetes. [Read the docs](doc/admin/executors/deploy_executors_kubernetes.md) for more information on executors on Kubernetes.
 
-This chart contains two deployments, Sourcegraph Executors and a private Docker Registry. It is a supplemental chart for the parent [sourcegraph/sourcegraph] Helm Chart if you wish to deploy executors
+## Native Kubernetes (k8s)
+This chart deploys executors that interact with the Kubernetes API to handle jobs. This is the preferred deployment option.  
+Your cluster will need to allow configuration of the following RBAC rules:
 
-Use cases:
+| API Groups | Resources          | Verbs                     | Reason                                                                                    |
+|------------|--------------------|---------------------------|-------------------------------------------------------------------------------------------|
+| `batch`    | `jobs`             | `create`, `delete`        | Executors create Job pods to run processes. Once Jobs are completed, they are cleaned up. |
+|            | `pods`, `pods/log` | `get`, `list`, `watch`    | Executors need to look up and steam logs from the Job Pods.                               |
 
-- Deploy Sourcegraph Executors on Kubernetes
-
-## Requirements
-
-* [Helm 3 CLI](https://helm.sh/docs/intro/install/)
-* Kubernetes 1.19 or greater
-
-## Installation
-
-Add the Sourcegraph charts repo to Helm:
-
-```sh
-helm repo add sourcegraph https://helm.sourcegraph.com/release
-```
-
-## Usage
-
-> The chart has to be installed in the same namespace as the parent [sourcegraph/sourcegraph] chart
-
-## Rendering manifests for kubectl deployment
-
-Manifests rendered using the `helm template` command can be used for direct deployment using `kubectl`.
-
-## Configuration Options
-
-Reference the table below for available configuration parameters and consult [executor] documentation.
-
-In addition to the documented values, the `executor` and `private-docker-registry` services also supports the following values
-
-- `executor.affinity` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity)
-- `executor.nodeSelector` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector)
-- `executor.tolerations` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/)
-- `executor.podSecurityContext` - [learn more](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod)
-- `executor.env` - consult `values.yaml`
-
-- `privateDockerRegistry.affinity` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity)
-- `privateDockerRegistry.nodeSelector` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector)
-- `privateDockerRegistry.tolerations` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/)
-- `privateDockerRegistry.podSecurityContext` - [learn more](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod)
-- `privateDockerRegistry.env` - consult `values.yaml` file
-
-| Key | Type | Default | Description |
-|-----|------|---------|-------------|
-| dind.image.registry | string | `"index.docker.io"` |  |
-| dind.image.repository | string | `"docker"` |  |
-| dind.image.tag | string | `"20.10.22-dind"` |  |
-| executor.enabled | bool | `true` |  |
-| executor.env.EXECUTOR_FRONTEND_PASSWORD | object | `{"value":""}` | The shared secret configured in the Sourcegraph instance site config under executors.accessToken. Required. |
-| executor.env.EXECUTOR_FRONTEND_URL | object | `{"value":""}` | The external URL of the Sourcegraph instance. Required. |
-| executor.env.EXECUTOR_QUEUE_NAME | object | `{"value":""}` | The name of the queue to pull jobs from to. Possible values: batches and codeintel. Required. |
-| executor.image.defaultTag | string | `"5.0.6@sha256:f78a0ba3d84941ec3577f19075e935bec34a9bfaa9dcb85908142fff017e9d6d"` |  |
-| executor.image.name | string | `"executor"` |  |
-| privateDockerRegistry.image.registry | string | `"index.docker.io"` |  |
-| privateDockerRegistry.image.repository | string | `"docker/regisry"` |  |
-| privateDockerRegistry.image.tag | int | `2` |  |
-| sourcegraph.affinity | object | `{}` | Affinity, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) |
-| sourcegraph.image.defaultTag | string | `"{{ .Chart.AppVersion }}"` | Global docker image tag |
-| sourcegraph.image.pullPolicy | string | `"IfNotPresent"` | Global docker image pull policy |
-| sourcegraph.image.repository | string | `"index.docker.io/sourcegraph"` | Global docker image registry or prefix |
-| sourcegraph.image.useGlobalTagAsDefault | bool | `false` | When set to true, sourcegraph.image.defaultTag is used as the default defaultTag for all services, instead of service-specific default defaultTags |
-| sourcegraph.imagePullSecrets | list | `[]` | Mount named secrets containing docker credentials |
-| sourcegraph.labels | object | `{}` | Add a global label to all resources |
-| sourcegraph.nameOverride | string | `""` | Set a custom name for the app.kubernetes.io/name annotation |
-| sourcegraph.nodeSelector | object | `{}` | NodeSelector, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector) |
-| sourcegraph.podAnnotations | object | `{}` | Add extra annotations to attach to all pods |
-| sourcegraph.podLabels | object | `{}` | Add extra labels to attach to all pods |
-| sourcegraph.tolerations | list | `[]` | Tolerations, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) |
-| storageClass.allowedTopologies | object | `{}` | Persistent volumes topology configuration, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/storage/storage-classes/#allowed-topologies) |
-| storageClass.create | bool | `false` | Enable creation of storageClass. Defaults to Google Cloud Platform. Disable if you have your own existing storage class |
-| storageClass.name | string | `"sourcegraph"` | Name of the storageClass. Use to customize to the existing storage class name |
-| storageClass.parameters | object | `{}` | Extra parameters of storageClass, consult your cloud provider persistent storage documentation |
-| storageClass.provisioner | string | `"kubernetes.io/gce-pd"` | Name of the storageClass provisioner, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/storage/storage-classes/#provisioner) and consult your cloud provider persistent storage documentation |
-| storageClass.type | string | `"pd-ssd"` | Value of `type` key in storageClass `parameters`, consult your cloud provider persistent storage documentation |
-
-## Troubleshooting
-
-See a list of running executor pods
-
-```sh
-kubectl get pods -l app=executor
-```
-
-Check logs of the executor container
-
-```sh
-kubectl logs -l app=executor -c executor
-```
-
-[sourcegraph/sourcegraph]: ../sourcegraph/
-[sourcegraph/sourcegraph-executor]: ./
-[executor]: https://docs.sourcegraph.com/admin/executors
+## Docker in Docker (dind)
+This chart deploys executors that deploy a [Docker in Docker](https://www.docker.com/blog/docker-can-now-run-within-docker/) sidecar with each executor pod to avoid accessing the host container runtime directly. This method requires privileged access to a container runtime daemon in order to operate correctly.  
+If you have security concerns, consider deploying via [a non-Kubernetes method](https://docs.sourcegraph.com/admin/executors).
diff --git a/charts/sourcegraph-executor/.helmignore → charts/sourcegraph-executor/dind/.helmignore b/charts/sourcegraph-executor/.helmignore → charts/sourcegraph-executor/dind/.helmignore
diff --git a/charts/sourcegraph-executor/dind/Chart.yaml b/charts/sourcegraph-executor/dind/Chart.yaml
@@ -0,0 +1,11 @@
+apiVersion: v2
+name: sourcegraph-executor-dind
+description: Chart for installing Sourcegraph executors using Docker in Docker (dind)
+icon: https://sourcegraph.com/favicon.ico
+type: application
+
+# Chart version, separate from Sourcegraph
+version: "5.0.2"
+
+# Version of Sourcegraph release
+appVersion: "5.0.2"
diff --git a/charts/sourcegraph-executor/dind/README.md b/charts/sourcegraph-executor/dind/README.md
@@ -0,0 +1,106 @@
+<!--
+  DO NOT EDIT README.md directly.
+  README.md is automatically generated from README.md.gotmpl
+-->
+
+# Sourcegraph Exexutor Helm Chart
+
+This chart contains two deployments, Sourcegraph Executors and a private Docker Registry. It is a supplemental chart for the parent [sourcegraph/sourcegraph] Helm Chart if you wish to deploy executors
+
+Use cases:
+
+- Deploy Sourcegraph Executors on Kubernetes
+
+## Requirements
+
+* [Helm 3 CLI](https://helm.sh/docs/intro/install/)
+* Kubernetes 1.19 or greater
+
+## Installation
+
+Add the Sourcegraph charts repo to Helm:
+
+```sh
+helm repo add sourcegraph https://helm.sourcegraph.com/release
+```
+
+## Usage
+
+> The chart has to be installed in the same namespace as the parent [sourcegraph/sourcegraph] chart
+
+## Rendering manifests for kubectl deployment
+
+Manifests rendered using the `helm template` command can be used for direct deployment using `kubectl`.
+
+## Configuration Options
+
+Reference the table below for available configuration parameters and consult [executor] documentation.
+
+In addition to the documented values, the `executor` and `private-docker-registry` services also supports the following values
+
+- `executor.affinity` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity)
+- `executor.nodeSelector` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector)
+- `executor.tolerations` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/)
+- `executor.podSecurityContext` - [learn more](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod)
+- `executor.env` - consult `values.yaml`
+
+- `privateDockerRegistry.affinity` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity)
+- `privateDockerRegistry.nodeSelector` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector)
+- `privateDockerRegistry.tolerations` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/)
+- `privateDockerRegistry.podSecurityContext` - [learn more](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod)
+- `privateDockerRegistry.env` - consult `values.yaml` file
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| dind.image.registry | string | `"index.docker.io"` |  |
+| dind.image.repository | string | `"docker"` |  |
+| dind.image.tag | string | `"20.10.22-dind"` |  |
+| executor.enabled | bool | `true` |  |
+| executor.env.EXECUTOR_FRONTEND_PASSWORD | object | `{"value":""}` | The shared secret configured in the Sourcegraph instance site config under executors.accessToken. Required. |
+| executor.env.EXECUTOR_FRONTEND_URL | object | `{"value":""}` | The external URL of the Sourcegraph instance. Required. |
+| executor.env.EXECUTOR_QUEUE_NAME | object | `{"value":""}` | The name of the queue to pull jobs from to. Possible values: batches and codeintel. **Either this or EXECUTOR_QUEUE_NAMES is required.** |
+| executor.env.EXECUTOR_QUEUE_NAMES | object | `{"value":""}` | The comma-separated list of names of multiple queues to pull jobs from to. Possible values: batches and codeintel. **Either this or EXECUTOR_QUEUE_NAME is required.** |
+| executor.image.defaultTag | string | `"5.1_230340_2023-06-23_5.0-93d39d620e83@sha256:172770133661d4d148327d2cac87c051a6409871ee6f6e28ce3495b60f883ad5"` |  |
+| executor.image.name | string | `"executor"` |  |
+| executor.replicaCount | int | `1` |  |
+| privateDockerRegistry.enabled | bool | `true` | Whether to deploy the private registry. Only one registry is needed when deploying multiple executors. More information: https://docs.sourcegraph.com/admin/executors/deploy_executors#using-private-registries |
+| privateDockerRegistry.image.registry | string | `"index.docker.io"` |  |
+| privateDockerRegistry.image.repository | string | `"docker/regisry"` |  |
+| privateDockerRegistry.image.tag | int | `2` |  |
+| privateDockerRegistry.storageSize | string | `"10Gi"` |  |
+| sourcegraph.affinity | object | `{}` | Affinity, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) |
+| sourcegraph.image.defaultTag | string | `"{{ .Chart.AppVersion }}"` | Global docker image tag |
+| sourcegraph.image.pullPolicy | string | `"IfNotPresent"` | Global docker image pull policy |
+| sourcegraph.image.repository | string | `"index.docker.io/sourcegraph"` | Global docker image registry or prefix |
+| sourcegraph.image.useGlobalTagAsDefault | bool | `false` | When set to true, sourcegraph.image.defaultTag is used as the default defaultTag for all services, instead of service-specific default defaultTags |
+| sourcegraph.imagePullSecrets | list | `[]` | Mount named secrets containing docker credentials |
+| sourcegraph.labels | object | `{}` | Add a global label to all resources |
+| sourcegraph.nameOverride | string | `""` | Set a custom name for the app.kubernetes.io/name annotation |
+| sourcegraph.nodeSelector | object | `{}` | NodeSelector, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector) |
+| sourcegraph.podAnnotations | object | `{}` | Add extra annotations to attach to all pods |
+| sourcegraph.podLabels | object | `{}` | Add extra labels to attach to all pods |
+| sourcegraph.tolerations | list | `[]` | Tolerations, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) |
+| storageClass.allowedTopologies | object | `{}` | Persistent volumes topology configuration, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/storage/storage-classes/#allowed-topologies) |
+| storageClass.create | bool | `false` | Enable creation of storageClass. Defaults to Google Cloud Platform. Disable if you have your own existing storage class |
+| storageClass.name | string | `"sourcegraph"` | Name of the storageClass. Use to customize to the existing storage class name |
+| storageClass.parameters | object | `{}` | Extra parameters of storageClass, consult your cloud provider persistent storage documentation |
+| storageClass.provisioner | string | `"kubernetes.io/gce-pd"` | Name of the storageClass provisioner, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/storage/storage-classes/#provisioner) and consult your cloud provider persistent storage documentation |
+| storageClass.type | string | `"pd-ssd"` | Value of `type` key in storageClass `parameters`, consult your cloud provider persistent storage documentation |
+
+## Troubleshooting
+
+See a list of running executor pods
+
+```sh
+kubectl get pods -l app=executor
+```
+
+Check logs of the executor container
+
+```sh
+kubectl logs -l app=executor -c executor
+```
+
+[sourcegraph/sourcegraph]: ../sourcegraph/
+[sourcegraph/sourcegraph-executor]: ./
+[executor]: https://docs.sourcegraph.com/admin/executors
diff --git a/charts/sourcegraph-executor/README.md.gotmpl → ...ourcegraph-executor/dind/README.md.gotmpl b/charts/sourcegraph-executor/README.md.gotmpl → ...ourcegraph-executor/dind/README.md.gotmpl