[DRAFT] EKS Hybrid Nodes Networking Docs (pod network routing, webhooks, mixed mode clusters)

latest/ug/nodes/hybrid-nodes-add-ons.adoc

@@ -62,7 +62,67 @@ The sections that follow describe differences between running compatible {aws} a
 [#hybrid-nodes-add-ons-core]
 == kube-proxy and CoreDNS
 
-EKS installs Kube-proxy and CoreDNS as self-managed add-ons by default when you create an EKS cluster with the {aws} API and {aws} SDKs, including from the {aws} CLI. You can overwrite these add-ons as Amazon EKS add-ons after cluster creation. Reference the EKS documentation for details on <<managing-kube-proxy>> and <<managing-coredns>>. If you are running a cluster with hybrid nodes and nodes in {aws} Cloud, we recommend that you have at least one CoreDNS replica on hybrid nodes and at least one CoreDNS replica on your nodes in {aws} Cloud.
+EKS installs kube-proxy and CoreDNS as self-managed add-ons by default when you create an EKS cluster with the {aws} API and {aws} SDKs, including from the {aws} CLI. You can overwrite these add-ons as Amazon EKS add-ons after cluster creation. Reference the EKS documentation for details on <<managing-kube-proxy>> and <<managing-coredns>>. 
+
+If you are running a mixed mode cluster with both hybrid nodes and nodes in {aws} Cloud, we recommend that you have at least one CoreDNS replica on hybrid nodes and at least one CoreDNS replica on your nodes in {aws} Cloud. CoreDNS can be configured such that your workloads will use the closest CoreDNS replica, meaning your cloud workloads will use the CoreDNS running in the cloud and your hybrid workloads will use the CoreDNS running on hybrid nodes. See the steps below for how to configure CoreDNS for a mixed mode cluster.
+
+. Add a topology zone label for each of your hybrid nodes, for example `topology.kubernetes.io/zone: onprem`. This can alternatively be done at the `nodeadm init` phase. Note, nodes running in {aws} Cloud automatically get a topology zone label applied to them.
++
+[source,bash,subs="verbatim,attributes,quotes"]
+----
+kubectl label node [.replaceable]`hybrid-node-name` topology.kubernetes.io/zone=[.replaceable]`zone`
+----
++
+. Add `podAntiAffinity` to the CoreDNS deployment configuration for the topology zone key. You can alternatively configure the CoreDNS deployment during installation with EKS add-ons.
++
+[source,bash,subs="verbatim,attributes,quotes"]
+----
+kubectl edit deployment coredns -n kube-system
+----
++
+[source,yaml,subs="verbatim,attributes"]
+----
+spec:
+  template:
+    spec:
+      affinity
+       ...
+        podAntiAffinity:
+          preferredDuringSchedulingIgnoredDuringExecution:
+          - podAffinityTerm:
+              labelSelector:
+                matchExpressions:
+                - key: k8s-app
+                  operator: In
+                  values:
+                  - kube-dns
+              topologyKey: kubernetes.io/hostname
+            weight: 100
+          - podAffinityTerm:
+              labelSelector:
+                matchExpressions:
+                - key: k8s-app
+                  operator: In
+                  values:
+                  - kube-dns
+              topologyKey: topology.kubernetes.io/zone
+            weight: 50
+      ...
+----
++
+. Add `trafficDistribution` to the kube-dns Service configuration.
++
+[source,bash,subs="verbatim,attributes"]
+----
+kubectl edit service kube-dns -n kube-system
+----
++
+[source,yaml,subs="verbatim,attributes"]
+----
+spec:
+  ...
+  trafficDistribution: PreferClose
+----
 
 [#hybrid-nodes-add-ons-cw]
 == CloudWatch Observability agent
@@ -71,7 +131,7 @@ As the CloudWatch Observability agent runs https://kubernetes.io/docs/reference/
 Node-level metrics are not available for hybrid nodes because link:AmazonCloudWatch/latest/monitoring/ContainerInsights.html[CloudWatch Container Insights,type="documentation"] depends on the availability of link:AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html[Instance Metadata Service,type="documentation"] (IMDS) for node-level metrics. Cluster, workload, pod, and container-level metrics are available for hybrid nodes.
 
 After installing the add-on by following the steps described in link:AmazonCloudWatch/latest/monitoring/install-CloudWatch-Observability-EKS-addon.html[Install the CloudWatch agent with the Amazon CloudWatch Observability,type="documentation"], the add-on manifest must be updated before the agent can run successfully on hybrid nodes. Edit the `amazoncloudwatchagents` resource on the cluster to add the `RUN_WITH_IRSA` environment variable as shown below.
-[source,yaml,subs="verbatim,attributes"]
+[source,bash,subs="verbatim,attributes"]
 ----
 kubectl edit amazoncloudwatchagents -n amazon-cloudwatch cloudwatch-agent
 ----