Skip to content

Latest commit

 

History

History
117 lines (92 loc) · 3.71 KB

README.md

File metadata and controls

117 lines (92 loc) · 3.71 KB
Raw

KubeRay Operator

This document provides instructions to install both CRDs (RayCluster, RayJob, RayService) and KubeRay operator with a Helm chart.

Helm

Make sure the version of Helm is v3+. Currently, existing CI tests are based on Helm v3.4.1 and v3.9.4.

helm version

Install CRDs and KubeRay operator

  • Install a stable version via Helm repository (only supports KubeRay v0.4.0+)

    helm repo add kuberay https://ray-project.github.io/kuberay-helm/

# Install both CRDs and KubeRay operator v0.5.0.
helm install kuberay-operator kuberay/kuberay-operator --version 0.5.0

# Check the KubeRay operator Pod in `default` namespace
kubectl get pods
# NAME                                READY   STATUS    RESTARTS   AGE
# kuberay-operator-6fcbb94f64-mbfnr   1/1     Running   0          17s

  • Install the nightly version

    # Step1: Clone KubeRay repository

# Step2: Move to `helm-chart/kuberay-operator`

# Step3: Install KubeRay operator
helm install kuberay-operator .

  • Install KubeRay operator without installing CRDs

    • In some cases, the installation of the CRDs and the installation of the operator may require different levels of admin permissions, so these two installations could be handled as different steps by different roles.
    • Use Helm's built-in --skip-crds flag to install the operator only. See this document for more details.
    # Step 1: Install CRDs only (for cluster admin)
kubectl create -k "github.com/ray-project/kuberay/manifests/cluster-scope-resources?ref=v0.5.0&timeout=90s"

# Step 2: Install KubeRay operator only. (for developer)
helm install kuberay-operator kuberay/kuberay-operator --version 0.5.0 --skip-crds

List the chart

To list the my-release deployment:

helm ls
# NAME                    NAMESPACE       REVISION        UPDATED                                   STATUS          CHART                   # APP VERSION
# kuberay-operator        default         1               2022-12-02 02:13:37.514445313 +0000 UTC   deployed        kuberay-operator-0.5.0  1.0

Uninstall the Chart

# Uninstall the `kuberay-operator` release
helm uninstall kuberay-operator

# The operator Pod should be removed.
kubectl get pods
# No resources found in default namespace.

Working with Argo CD

If you are using Argo CD to manage the operator, you will encounter the issue which complains the CRDs too long. Same with this issue. The recommended solution is to split the operator into two Argo apps, such as:

  • The first app just for installing the CRDs with Replace=true directly, snippet:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ray-operator-crds
spec:
  project: default
  source:
    repoURL: https://github.com/ray-project/kuberay
    targetRevision: v0.5.0
    path: helm-chart/kuberay-operator/crds
  destination:
    server: https://kubernetes.default.svc
  syncPolicy:
    syncOptions:
    - Replace=true
...
  • The second app that installs the Helm chart with skipCrds=true (new feature in Argo CD 2.3.0), snippet:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ray-operator
spec:
  source:
    repoURL: https://github.com/ray-project/kuberay
    targetRevision: v0.5.0
    path: helm-chart/kuberay-operator
    helm:
      skipCrds: true
  destination:
    server: https://kubernetes.default.svc
    namespace: ray-operator
  syncPolicy:
    syncOptions:
    - CreateNamespace=true
...