Executors: add native k8s executors

[sanderginn]

Description

This PR adds a chart for deploying native k8s executors. It also polishes the dind implementation.

Checklist

• Follow the manual testing process
• Update changelog

Test plan

• Ran the test steps as described in https://github.com/sourcegraph/sourcegraph/issues/50824
• Unit tests

[sanderginn]

Here's an example override.yaml that I used to deploy executors with to the test cluster (in this case, a batches executor).

I created a dedicated node with the label sourcegraph-executors=true in order to target it with the node selector. This is required because the executor spawns a job that mounts the same PVC as the executor deployment, so they must reside on the same node. I will propose the addition of setting pod affinity on the jobs, because the current implementation doesn't support pain-free node autoscaling.

executor:
  queueName: batches
  nodeSelector:
    sourcegraph-executors: "true"
  env:
    EXECUTOR_FRONTEND_URL:
      value: "http://sourcegraph-frontend:30080"
    # -- The shared secret configured in the Sourcegraph instance site config under executors.accessToken. Required.
    EXECUTOR_FRONTEND_PASSWORD:
      value: "hunter2hunter2hunter2hunter2"
    # -- The name of the queue to pull jobs from to. Possible values: batches and codeintel. Required.
    EXECUTOR_QUEUE_NAME:
      value: "batches"
    EXECUTOR_MAXIMUM_NUM_JOBS:
      value: "10"
    SRC_LOG_LEVEL:
      value: info
    SRC_LOG_FORMAT:
      value: condensed
    SRC_TRACE_LOG:
      value: "false"
    EXECUTOR_KUBERNETES_RESOURCE_REQUEST_MEMORY:
      value: 1Gi
    # For local deployments the host is 'host.docker.internal' and this needs to be true
    EXECUTOR_DOCKER_ADD_HOST_GATEWAY:
      value: "false"
    # Useful for debugging.
    EXECUTOR_KUBERNETES_NAMESPACE:
      value: sourcegraph
    KUBERNETES_KEEP_JOBS:
      value: "true"
    EXECUTOR_KEEP_WORKSPACES:
      value: "true"

[Piszmog]

Overall makes sense (first time I am seeing this syntax). The only change is that we do not need the private docker registry for the native k8s options. Since that doesn't do anything with docker directly but relies on kubernetes to do the creating of pods/containers

[sanderginn]

@Piszmog resolved the issues, I think it's a-okay for merging now

[jac]

Hey @sanderginn
I'm pretty sure this will break the helm chart publishing as the publishing requires charts/sourcegraph-executor/Chart.yaml to exist

deploy-sourcegraph-helm/.github/workflows/gcs_chart_publish_release.yml

Lines 47 to 48 in 7cdad50

	- name: Package helm charts
	run: for i in charts/*; do helm package -u $i; done

Additionally in this PR both k8s and dind Chart.yaml files specify the name as sourcegraph-executor which will conflict as the output of helm package is <name>-<version>.tar.gz

[Piszmog]

Just one more minor change

[sanderginn]

@jac thanks for that catch 👍
I've added a separate step for packaging the executor charts.

I've also changed the names of the charts. I've suffixed them with the implementation. This means that the dind chart, that used to be published as sourcegraph-executor, is now published as sourcegraph-executor-dind. Do you think this is an issue, as customers with the installed chart will have to use the new chart name to upgrade? (I don't know if there are many customers with this deployed, if any, tbh). I can revert the dind one to sourcegraph-executor if you think that's better.

[jac]

I'm not sure how many customers have it deployed either.
As long as it is noted in the changelog I think it should be fine

[sanderginn]

👍 thanks. Added a changelog