rancher · btat · Dec 6, 2024 · Dec 6, 2024
@@ -141,7 +141,7 @@ The config.Node golang struct is defined https://github.com/k3s-io/k3s/blob/mast
 [IMPORTANT]
 .Version Gate
 ====
-Available as of September 2023 releases: v1.24.17+k3s1, v1.25.13+k3s1, v1.26.8+k3s1, v1.27.5+k3s1, v1.28.1+k3s1
+Available as of the September 2023 releases: v1.24.17+k3s1, v1.25.13+k3s1, v1.26.8+k3s1, v1.27.5+k3s1, v1.28.1+k3s1
 ====
 
 

@@ -195,7 +195,7 @@ List bootstrap tokens, showing their ID, description, and remaining time-to-live
 [IMPORTANT]
 .Version Gate
 ====
-Available as of 2023-10 releases (v1.28.2+k3s1, v1.27.7+k3s1, v1.26.10+k3s1, v1.25.15+k3s1).
+Available as of the October 2023 releases (v1.28.2+k3s1, v1.27.7+k3s1, v1.26.10+k3s1, v1.25.15+k3s1).
 ====
 
 

@@ -206,7 +206,7 @@ When using a high-availability setup of 3 server nodes, the number of agents can
 
 It is recommended to join agent nodes in batches of 50 or less to allow the CPU to free up space, as there is a spike on node join. Remember to modify the default `cluster-cidr` if desiring more than 255 nodes!
 
-xref:reference/resource-profiling.adoc#server-sizing-requirements-for-k3s[Resource Profiling] contains more information how these recommendations were found.
+xref:reference/resource-profiling.adoc#_server_sizing_requirements_for_k3s[Resource Profiling] contains more information how these recommendations were found.
 
 == Networking
 

@@ -2,13 +2,15 @@
 
 [CAUTION]
 ====
-Uninstalling K3s deletes the local cluster data, configuration, and all of the scripts and CLI tools. +
-It does not remove any data from external datastores, or created by pods using external Kubernetes storage volumes.
+Uninstalling K3s may cause data loss!
 ====
 
-
 If you installed K3s using the installation script, a script to uninstall K3s was generated during installation.
 
+Running the uninstall script stops K3s and all running pods, and deletes the local cluster datastore, xref:storage.adoc#_setting_up_the_local_storage_provider[Local Storage] Persistent Volume data, node configuration, and all of the scripts and CLI tools.
+
+It does not remove any data from external datastores, or created by pods using external Kubernetes Persistent Volumes.
+
 If you are planning on rejoining a node to an existing cluster after uninstalling and reinstalling, be sure to delete the node from the cluster to ensure that the node password secret is removed. See the xref:architecture.adoc#_how_agent_node_registration_works[Node Registration] documentation for more information.
 
 == Uninstalling Servers

@@ -2,20 +2,28 @@
 
 https://github.com/k8snetworkplumbingwg/multus-cni[Multus CNI] is a CNI plugin that enables attaching multiple network interfaces to pods. Multus does not replace CNI plugins, instead it acts as a CNI plugin multiplexer. Multus is useful in certain use cases, especially when pods are network intensive and require extra network interfaces that support dataplane acceleration techniques such as SR-IOV.
 
+For more information about Multus, refer to the https://github.com/k8snetworkplumbingwg/multus-cni/tree/master/docs[multus-cni] documentation.
+
 Multus can not be deployed standalone. It always requires at least one conventional CNI plugin that fulfills the Kubernetes cluster network requirements. That CNI plugin becomes the default for Multus, and will be used to provide the primary interface for all pods. When deploying K3s with default options, that CNI plugin is Flannel.
 
-== Add the Helm Repository
+[NOTE]
+.Version Gate
+====
+K3s uses a fixed CNI binary path as of the October 2024 releases: v1.28.15+k3s1, v1.29.10+k3s1, v1.30.6+k3s1, v1.31.2+k3s1.
+====
 
-To deploy Multus, we recommend using the following helm repo:
+K3s looks at `$DATA_DIR/data/cni` for CNI plugin binaries. By default this is `/var/lib/rancher/k3s/data/cni`. Additional CNI plugins should be installed to this location.
 
-----
-helm repo add rke2-charts https://rke2-charts.rancher.io
-helm repo update
-----
+Prior to the October 2024 releases, CNI binaries were part of the K3s userspace bundle at `$DATA_DIR/data/$HASH/bin`, where the hash is unique to each release of K3s.
+This made it difficult to deploy additional CNI plugins, as the path would change every time K3s was upgraded.
 
-== Configure the IPAM plugin
+If deploying Multus to an older release of K3s, you should use `/var/lib/rancher/k3s/data/current/bin/` as the CNI bin dir, but expect that the plugins will need to be re-deployed whenever K3s is upgraded.
 
-An IPAM plugin is required to assign IP addresses on the extra interfaces created by Multus.
+== Deploy with an IPAM plugin
+
+An IP Address Manager (IPAM) plugin is required to assign IP addresses on the extra interfaces created by Multus. One or more IPAMs can be installed; the examples below each show use of a single IPAM plugin but they may be combined as needed.
+
+The helm deployment examples below will deploy a DaemonSet to create Multus pods to install the required CNI binaries in `/var/lib/rancher/k3s/data/cni/` and Multus CNI config in `/var/lib/rancher/k3s/agent/etc/cni/net.d`.
 
 [tabs,sync-group-id=MultusIPAMplugins]
 ======
@@ -24,15 +32,26 @@ host-local::
 --
 The host-local IPAM plugin allocates ip addresses out of a set of address ranges. It stores the state locally on the host filesystem, hence ensuring uniqueness of IP addresses on a single host. Therefore, we don't recommend it for multi-node clusters. This IPAM plugin does not require any extra deployment. For more information: https://www.cni.dev/plugins/current/ipam/host-local/.
 
-To use the host-local plugin, please create a file called `multus-values.yaml` with the following content:
+To use the host-local plugin, deploy Multus with the following configuration:
 
 [,yaml]
 ----
-config:
-  cni_conf:
-    confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
-    binDir: /var/lib/rancher/k3s/data/current/bin/
-    kubeconfig: /var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig
+apiVersion: helm.cattle.io/v1
+kind: HelmChart
+metadata:
+  name: multus
+  namespace: kube-system
+spec:
+  repo: https://rke2-charts.rancher.io
+  chart: rke2-multus
+  targetNamespace: kube-system
+  valuesContent: |-
+    config:
+      fullnameOverride: multus
+      cni_conf:
+        confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
+        binDir: /var/lib/rancher/k3s/data/cni/
+        kubeconfig: /var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig
 ----
 --
 
@@ -41,21 +60,58 @@ Whereabouts::
 --
 https://github.com/k8snetworkplumbingwg/whereabouts[Whereabouts] is an IP Address Management (IPAM) CNI plugin that assigns IP addresses cluster-wide.
 
-To use the Whereabouts IPAM plugin, please create a file called `multus-values.yaml` with the following content:
+To use the Whereabouts IPAM plugin, deploy Multus with the following configuration:
 
 [,yaml]
 ----
-config:
-  cni_conf:
-    confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
-    binDir: /var/lib/rancher/k3s/data/current/bin/
-    kubeconfig: /var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig
-rke2-whereabouts:
-  fullnameOverride: whereabouts
-  enabled: true
-  cniConf:
-    confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
-    binDir: /var/lib/rancher/k3s/data/current/bin/
+apiVersion: helm.cattle.io/v1
+kind: HelmChart
+metadata:
+  name: multus
+  namespace: kube-system
+spec:
+  repo: https://rke2-charts.rancher.io
+  chart: rke2-multus
+  targetNamespace: kube-system
+  valuesContent: |-
+    config:
+      fullnameOverride: multus
+      cni_conf:
+        confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
+        binDir: /var/lib/rancher/k3s/data/cni/
+        kubeconfig: /var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig
+    rke2-whereabouts:
+      fullnameOverride: whereabouts
+      enabled: true
+      cniConf:
+        confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
+        binDir: /var/lib/rancher/k3s/data/cni/
+----
+
+When using whereabouts on K3s, `configuration_path` must be set to `/var/lib/rancher/k3s/agent/etc/cni/net.d/whereabouts.d/whereabouts.conf` in the NetworkAttachmentDefinition's `ipam` configuration.
+
+For example, when using whereabouts as the IPAM with the macvlan plugin:
+
+[,yaml]
+----
+apiVersion: k8s.cni.cncf.io/v1
+kind: NetworkAttachmentDefinition
+metadata:
+  name: macvlan-whereabouts
+spec:
+  config: |-
+    {
+      "cniVersion": "1.0.0",
+      "type": "macvlan",
+      "master": "eth0",
+      "mode": "bridge",
+      "ipam": {
+        "type": "whereabouts",
+        "range": "172.17.0.0/24",
+        "gateway": "172.17.0.1",
+        "configuration_path": "/var/lib/rancher/k3s/agent/etc/cni/net.d/whereabouts.d/whereabouts.conf"
+      }
+    }
 ----
 --
 
@@ -64,29 +120,64 @@ Multus DHCP daemon::
 --
 The dhcp IPAM plugin can be deployed when there is already a DHCP server running on the network. This daemonset takes care of periodically renewing the DHCP lease. For more information please check the official docs of https://www.cni.dev/plugins/current/ipam/dhcp/[DHCP IPAM plugin].
 
-To use the DHCP plugin, please create a file called `multus-values.yaml` with the following content:
+To use the DHCP plugin, deploy Multus with the following configuration:
 
 [,yaml]
 ----
-config:
-  cni_conf:
-    confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
-    binDir: /var/lib/rancher/k3s/data/current/bin/
-    kubeconfig: /var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig
-manifests:
-  dhcpDaemonSet: true
+apiVersion: helm.cattle.io/v1
+kind: HelmChart
+metadata:
+  name: multus
+  namespace: kube-system
+spec:
+  repo: https://rke2-charts.rancher.io
+  chart: rke2-multus
+  targetNamespace: kube-system
+  valuesContent: |-
+    config:
+      fullnameOverride: multus
+      cni_conf:
+        confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
+        binDir: /var/lib/rancher/k3s/data/cni/
+        kubeconfig: /var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig
+    manifests:
+      dhcpDaemonSet: true
 ----
 --
 ======
 
-== Deploy Multus
+=== Using Multus
 
-After creating the `multus-values.yaml` file, everything is ready to install Multus:
+Once Multus has been deployed, you can create NetworkAttachmentDefinition resources, and reference these in Pod specs to attach additional interfaces.
+For example, using the whereabouts example above, you can create an `eth1` interface on a Pod using the `k8s.v1.cni.cncf.io/networks` annotation:
 
+[,yaml]
 ----
-helm install multus rke2-charts/rke2-multus -n kube-system --kubeconfig /etc/rancher/k3s/k3s.yaml --values multus-values.yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: multus-demo
+  labels:
+    app: multus-demo
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: multus-demo
+  template:
+    metadata:
+      annotations:
+        k8s.v1.cni.cncf.io/networks: macvlan-whereabouts@eth1
+      labels:
+        app: multus-demo
+    spec:
+      containers:
+      - name: shell
+        image: docker.io/rancher/mirrored-library-busybox:1.36.1
+        imagePullPolicy: IfNotPresent
+        command:
+          - sleep
+          - "3600"
 ----
 
-The helm chart install will deploy a DaemonSet to create Multus pods for installing the required CNI binaries in `/var/lib/rancher/k3s/data/current/` and Multus CNI config in `/var/lib/rancher/k3s/agent/etc/cni/net.d`.
-
-For more information about Multus, refer to the https://github.com/k8snetworkplumbingwg/multus-cni/tree/master/docs[multus-cni] documentation.
+See the upstream documentation for additional information and examples.
@@ -102,7 +102,7 @@ In order to reduce binary size, K3s removes all "in-tree" (built-in) cloud provi
 * Hosts the ServiceLB LoadBalancer controller.
 * Clears the `node.cloudprovider.kubernetes.io/uninitialized` taint that is present when the cloud-provider is set to `external`
 
-Before deploying an external CCM, you must start all K3s servers with the `--disable-cloud-controller` flag to disable to embedded CCM.
+Before deploying an external CCM, you must start all K3s servers with the `--disable-cloud-controller` flag to disable the embedded CCM.
 
 [NOTE]
 ====

@@ -190,7 +190,7 @@ OPTIONS:
 [IMPORTANT]
 .バージョンゲート
 ====
-2023-10 リリース（v1.28.2+k3s1、v1.27.7+k3s1、v1.26.10+k3s1、v1.25.15+k3s1）から利用可能です。
+2023年10月のリリース（v1.28.2+k3s1、v1.27.7+k3s1、v1.26.10+k3s1、v1.25.15+k3s1）から利用可能です。
 ====
 
 

@@ -207,7 +207,7 @@ When using a high-availability setup of 3 server nodes, the number of agents can
 
 It is recommended to join agent nodes in batches of 50 or less to allow the CPU to free up space, as there is a spike on node join. Remember to modify the default `cluster-cidr` if desiring more than 255 nodes!
 
-xref:reference/resource-profiling.adoc#server-sizing-requirements-for-k3s[Resource Profiling] contains more information how these recommendations were found.
+xref:reference/resource-profiling.adoc#_server_sizing_requirements_for_k3s[Resource Profiling] contains more information how these recommendations were found.
 
 == ネットワーキング
 

@@ -2,13 +2,16 @@
 
 [CAUTION]
 ====
-K3sをアンインストールすると、ローカルクラスターのデータ、設定、およびすべてのスクリプトとCLIツールが削除されます。 +
-外部データストアからのデータや、外部Kubernetesストレージボリュームを使用してポッドによって作成されたデータは削除されません。
+Uninstalling K3s may cause data loss!
 ====
 
-
 インストールスクリプトを使用してK3sをインストールした場合、インストール中にK3sをアンインストールするためのスクリプトが生成されました。
 
+Running the uninstall script stops K3s and all running pods, and deletes the local cluster datastore, xref:storage.adoc#_setting_up_the_local_storage_provider[Local Storage] Persistent Volume data, node configuration, and all of the scripts and CLI tools.
+
+It does not remove any data from external datastores, or created by pods using external Kubernetes Persistent Volumes.
+
+
 アンインストール後にノードを既存のクラスターに再参加させる予定がある場合は、ノードのパスワードシークレットが削除されるように、クラスターからノードを削除することを忘れないでください。詳細については、xref:architecture.adoc#_how_agent_node_registration_works[ノード登録]のドキュメントを参照してください。
 
 == サーバーのアンインストール