From 55de548345de176926f7b817f15ba5aef2678ba4 Mon Sep 17 00:00:00 2001
From: David-VTUK <david.holder@gmail.com>
Date: Mon, 9 Dec 2024 13:35:06 +0000
Subject: [PATCH] Deployed 661b271 with MkDocs version: 1.6.1

---
 index.html               | 4 ++--
 search/search_index.json | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/index.html b/index.html
index e342581..ecd9eb7 100644
--- a/index.html
+++ b/index.html
@@ -123,7 +123,7 @@ <h2 id="lab-environments">Lab Environments</h2>
 <p><a href="https://kubernetes.io/docs/setup/learning-environment/minikube/">Minikube Setup</a></p>
 <p><a href="https://kubernetes.io/blog/2020/05/21/wsl-docker-kubernetes-on-the-windows-desktop/">WSL Docker Kubernetes on the Windows Desktop</a></p>
 <h2 id="contact">Contact</h2>
-<p>Twitter = @VT_UK</p>
+<p>Twitter/X = <a href="https://x.com/VT_UK">@VT_UK</a></p>
               
             </div>
           </div><footer>
@@ -172,5 +172,5 @@ <h2 id="contact">Contact</h2>
 
 <!--
 MkDocs version : 1.6.1
-Build Date UTC : 2024-12-09 13:30:49.233113+00:00
+Build Date UTC : 2024-12-09 13:35:06.371908+00:00
 -->
diff --git a/search/search_index.json b/search/search_index.json
index 24f14c0..01ebe19 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"CKA-Study Guide The following guide is based on the Certified Kubernetes Administrator Exam Curriculum 1.29 and consists of the following sections: Revision Topics Notes from revising the topics listed in the curriculum, grouped to correlate to the various sections listed in the curriculum. Lab Guide With Example Answers To complement the revision topics, these lab guides are designed to exercise the topics learnt from the corresponding revision topics sections. Resources / References http://www.kubernet.io/ (Currently offline) https://github.com/kelseyhightower/kubernetes-the-hard-way Certified Kubernetes Administrator Course on A Cloud Guru https://kubernetes.io/docs/ https://kubernetes.io/docs/tasks/ Lab Environments Minikube Setup WSL Docker Kubernetes on the Windows Desktop Contact Twitter = @VT_UK","title":"Home"},{"location":"#cka-study-guide","text":"The following guide is based on the Certified Kubernetes Administrator Exam Curriculum 1.29 and consists of the following sections:","title":"CKA-Study Guide"},{"location":"#revision-topics","text":"Notes from revising the topics listed in the curriculum, grouped to correlate to the various sections listed in the curriculum.","title":"Revision Topics"},{"location":"#lab-guide-with-example-answers","text":"To complement the revision topics, these lab guides are designed to exercise the topics learnt from the corresponding revision topics sections.","title":"Lab Guide With Example Answers"},{"location":"#resources-references","text":"http://www.kubernet.io/ (Currently offline) https://github.com/kelseyhightower/kubernetes-the-hard-way Certified Kubernetes Administrator Course on A Cloud Guru https://kubernetes.io/docs/ https://kubernetes.io/docs/tasks/","title":"Resources / References"},{"location":"#lab-environments","text":"Minikube Setup WSL Docker Kubernetes on the Windows Desktop","title":"Lab Environments"},{"location":"#contact","text":"Twitter = @VT_UK","title":"Contact"},{"location":"lab-guide/00-general-advice/","text":"General Advice Imperative vs Declarative The Lab guide will present both imperative and declarative answers to questions where possible. Due to time constraints in the exam, using imperative commands may be more efficient. Outside of the exam, declarative management is the preferred approach. Use the documentation At time of writing, Kubernetes Documentation , Kubernetes GitHub and Kubernetes Blog are permitted to be used during the exam as per the current guidelines . If you need to get boilerplate YAML manifests to address an exam question, feel free to get them from here. The key is to know what to search for. For example, if asked to create and apply a service object, search for Service YAML in the Kubernetes docs and modify an example to suit the question objectives. Caution on documentation When using the Kubernetes.io docs, be careful of the search results - some will point to resources outside of kubernetes.io and the list of accepted resource for the exam. Should you navigate to these your exam may be cancelled. Sanity check the URL the search result is pointing to prior to procedding. Don't write YAML manifests from scratch In addition to the above example, leverage the combination of --dry-run and -o to generate YAML that can be modified. kubectl run nginx --image=nginx --dry-run=client -o yaml > pod.yaml Which will generate: apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: nginx name: nginx spec: containers: - image: nginx name: nginx resources: {} dnsPolicy: ClusterFirst restartPolicy: Always status: {} --dry-run won't commit the command -o yaml specifies YAML as the output > pod.yaml redirects the output to a file Become BFF's with kubectl Effective use of kubectl will be extremely important to pass the exam. Become very comfortable with it. Unsure of an objects spec, type, options? kubectl explain <resourcetype>","title":"Part Zero - General Advice"},{"location":"lab-guide/00-general-advice/#general-advice","text":"","title":"General Advice"},{"location":"lab-guide/00-general-advice/#imperative-vs-declarative","text":"The Lab guide will present both imperative and declarative answers to questions where possible. Due to time constraints in the exam, using imperative commands may be more efficient. Outside of the exam, declarative management is the preferred approach.","title":"Imperative vs Declarative"},{"location":"lab-guide/00-general-advice/#use-the-documentation","text":"At time of writing, Kubernetes Documentation , Kubernetes GitHub and Kubernetes Blog are permitted to be used during the exam as per the current guidelines . If you need to get boilerplate YAML manifests to address an exam question, feel free to get them from here. The key is to know what to search for. For example, if asked to create and apply a service object, search for Service YAML in the Kubernetes docs and modify an example to suit the question objectives.","title":"Use the documentation"},{"location":"lab-guide/00-general-advice/#caution-on-documentation","text":"When using the Kubernetes.io docs, be careful of the search results - some will point to resources outside of kubernetes.io and the list of accepted resource for the exam. Should you navigate to these your exam may be cancelled. Sanity check the URL the search result is pointing to prior to procedding.","title":"Caution on documentation"},{"location":"lab-guide/00-general-advice/#dont-write-yaml-manifests-from-scratch","text":"In addition to the above example, leverage the combination of --dry-run and -o to generate YAML that can be modified. kubectl run nginx --image=nginx --dry-run=client -o yaml > pod.yaml Which will generate: apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: nginx name: nginx spec: containers: - image: nginx name: nginx resources: {} dnsPolicy: ClusterFirst restartPolicy: Always status: {} --dry-run won't commit the command -o yaml specifies YAML as the output > pod.yaml redirects the output to a file","title":"Don't write YAML manifests from scratch"},{"location":"lab-guide/00-general-advice/#become-bffs-with-kubectl","text":"Effective use of kubectl will be extremely important to pass the exam. Become very comfortable with it.","title":"Become BFF's with kubectl"},{"location":"lab-guide/00-general-advice/#unsure-of-an-objects-spec-type-options","text":"kubectl explain <resourcetype>","title":"Unsure of an objects spec, type, options?"},{"location":"lab-guide/01-architcture-installation-configuration/","text":"Lab Exercises for Cluster Architecture, Installation and Configuration Exercise 0 - Setup Prepare a cluster (Single node, kubeadm, k3s, etc) Prepare two vanilla VM's (No Kubernetes components installed) with the kubeadm binary installed (feel free to do this beforehand, I doubt it will be a requirement to add apt sources to get it) Open browser tabs to Kubernetes Documentation , Kubernetes GitHub and Kubernetes Blog (these are permitted as per the current guidelines ) Exercise 1 - RBAC A third party application requires access to describe job objects that reside in a namespace called rbac . Perform the following: Create a namespace called rbac Create a service account called job-inspector for the rbac namespace Create a role that has rules to get and list job objects Create a rolebinding that binds the service account job-inspector to the role created in step 3 Prove the job-inspector service account can \"get\" job objects but not deployment objects Answer - Imperative kubectl create namespace rbac kubectl create sa job-inspector -n rbac kubectl create role job-inspector --verb=get --verb=list --resource=jobs -n rbac kubectl create rolebinding permit-job-inspector --role=job-inspector --serviceaccount=rbac:job-inspector -n rbac kubectl --as=system:serviceaccount:rbac:job-inspector auth can-i get job -n rbac kubectl --as=system:serviceaccount:rbac:job-inspector auth can-i get deployment -n rbac Answer - Declarative apiVersion: v1 kind: Namespace metadata: name: rbac --- apiVersion: v1 kind: ServiceAccount metadata: name: job-inspector namespace: rbac --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: job-inspector namespace: rbac rules: - apiGroups: [\"batch\"] resources: [\"jobs\"] verbs: [\"get\", \"list\"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: permit-job-inspector namespace: rbac roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: job-inspector subjects: - kind: ServiceAccount name: job-inspector namespace: rbac Exercise 2 - Use Kubeadm to install a basic cluster On a node, install kubeadm and stand up the control plane, using 10.244.0.0/16 as the pod network CIDR, and kube-flannel.yml as the CNI On a node, install kubeadm and join it to the cluster as a worker node Answer Node 1 Prep kubeadm (as mentioned above, I doubt we will need to do this part in the exam) # Install a container runtime, IE https://github.com/containerd/containerd/blob/main/docs/getting-started.md sudo apt update && apt install containerd -y sudo apt-get update # apt-transport-https may be a dummy package; if so, you can skip that package sudo apt-get install -y apt-transport-https ca-certificates curl gpg # If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below. # sudo mkdir -p -m 755 /etc/apt/keyrings curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg # This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list sudo apt-get update sudo apt-get install -y kubelet kubeadm kubectl sudo apt-mark hold kubelet kubeadm kubectl You may need to execute the following depending on the underlying OS: modprobe br_netfilter echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables echo 1 > /proc/sys/net/ipv4/ip_forward Turn this node into a master sudo kubeadm init --pod-network-cidr=10.244.0.0/16 ... mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config ... kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml ... Note the join command, ie: kubeadm join 172.16.10.210:6443 --token 9tjntl.10plpxqy85g8a0ui \\ --discovery-token-ca-cert-hash sha256:381165c9a9f19a123bd0fee36fe36d15e918062dcc94711ff5b286ee1f86b92b Node 2: Run the join command taken from the previous step kubeadm join 172.16.10.210:6443 --token 9tjntl.10plpxqy85g8a0ui \\ --discovery-token-ca-cert-hash sha256:381165c9a9f19a123bd0fee36fe36d15e918062dcc94711ff5b286ee1f86b92b Validate by running kubectl get no on the master node: kubectl get no NAME STATUS ROLES AGE VERSION ubuntu Ready control-plane,master 9m53s v1.31.3 ubuntu2 Ready <none> 50s v1.31.3 Exercise 3 - Manage a highly-available Kubernetes cluster Using etcdctl , determine the health of the etcd cluster Using etcdctl , identify the list of members On the master node, determine the health of the cluster by probing the API endpoint Answer root@ip-172-31-31-80:~# ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \\ --cacert=/etc/kubernetes/pki/etcd/ca.crt \\ --cert=/etc/kubernetes/pki/etcd/server.crt \\ --key=/etc/kubernetes/pki/etcd/server.key \\ endpoint health https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 6.852757ms root@ip-172-31-31-80:~# ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \\ --cacert=/etc/kubernetes/pki/etcd/ca.crt \\ --cert=/etc/kubernetes/pki/etcd/server.crt \\ --key=/etc/kubernetes/pki/etcd/server.key \\ member list f90d942cc570125d, started, ip-172-31-31-80, https://172.31.31.80:2380, https://172.31.31.80:2379, false curl -k https://localhost:6443/healthz?verbose [+]ping ok [+]log ok [+]etcd ok [+]poststarthook/start-kube-apiserver-admission-initializer ok ... Exercise 4 - Perform a version upgrade on a Kubernetes cluster using Kubeadm Using kubeadm , upgrade a cluster to the lastest version Answer If held, unhold the kubeadm version sudo apt-mark unhold kubeadm Upgrade the kubeadm version: sudo apt-get install --only-upgrade kubeadm plan the upgrade: sudo kubeadm upgrade plan Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': COMPONENT CURRENT AVAILABLE kubelet 1 x v1.29.10 v1.31.3 Upgrade to the latest stable version: COMPONENT CURRENT AVAILABLE kube-apiserver v1.29.10 v1.31.3 kube-controller-manager v1.29.10 v1.31.3 kube-scheduler v1.29.10 v1.31.3 kube-proxy v1.29.10 v1.31.3 CoreDNS 1.7.0 1.11.3 etcd 3.4.9-1 3.5.15-0 Upgrade the cluster kubeadm upgrade apply v1.20.2 Upgrade Kubelet: sudo apt-get install --only-upgrade kubelet Exercise 5 - Implement etcd backup and restore Take a backup of etcd Verify the etcd backup has been successful Restore the backup back to the cluster Answer Take a snapshot of etcd: ETCDCTL_API=3 etcdctl snapshot save snapshot.db --cacert /etc/kubernetes/pki/etcd/server.crt --cert /etc/kubernetes/pki/etcd/ca.crt --key /etc/kubernetes/pki/etcd/ca.key Verify the snapshot: sudo ETCDCTL_API=3 etcdctl --write-out=table snapshot status snapshot.db Perform a restore: ETCDCTL_API=3 etcdctl snapshot restore snapshot.db","title":"Part One - Architecture, Installation and Configuration"},{"location":"lab-guide/01-architcture-installation-configuration/#lab-exercises-for-cluster-architecture-installation-and-configuration","text":"","title":"Lab Exercises for Cluster Architecture, Installation and Configuration"},{"location":"lab-guide/01-architcture-installation-configuration/#exercise-0-setup","text":"Prepare a cluster (Single node, kubeadm, k3s, etc) Prepare two vanilla VM's (No Kubernetes components installed) with the kubeadm binary installed (feel free to do this beforehand, I doubt it will be a requirement to add apt sources to get it) Open browser tabs to Kubernetes Documentation , Kubernetes GitHub and Kubernetes Blog (these are permitted as per the current guidelines )","title":"Exercise 0 - Setup"},{"location":"lab-guide/01-architcture-installation-configuration/#exercise-1-rbac","text":"A third party application requires access to describe job objects that reside in a namespace called rbac . Perform the following: Create a namespace called rbac Create a service account called job-inspector for the rbac namespace Create a role that has rules to get and list job objects Create a rolebinding that binds the service account job-inspector to the role created in step 3 Prove the job-inspector service account can \"get\" job objects but not deployment objects Answer - Imperative kubectl create namespace rbac kubectl create sa job-inspector -n rbac kubectl create role job-inspector --verb=get --verb=list --resource=jobs -n rbac kubectl create rolebinding permit-job-inspector --role=job-inspector --serviceaccount=rbac:job-inspector -n rbac kubectl --as=system:serviceaccount:rbac:job-inspector auth can-i get job -n rbac kubectl --as=system:serviceaccount:rbac:job-inspector auth can-i get deployment -n rbac Answer - Declarative apiVersion: v1 kind: Namespace metadata: name: rbac --- apiVersion: v1 kind: ServiceAccount metadata: name: job-inspector namespace: rbac --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: job-inspector namespace: rbac rules: - apiGroups: [\"batch\"] resources: [\"jobs\"] verbs: [\"get\", \"list\"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: permit-job-inspector namespace: rbac roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: job-inspector subjects: - kind: ServiceAccount name: job-inspector namespace: rbac","title":"Exercise 1 - RBAC"},{"location":"lab-guide/01-architcture-installation-configuration/#exercise-2-use-kubeadm-to-install-a-basic-cluster","text":"On a node, install kubeadm and stand up the control plane, using 10.244.0.0/16 as the pod network CIDR, and kube-flannel.yml as the CNI On a node, install kubeadm and join it to the cluster as a worker node Answer Node 1 Prep kubeadm (as mentioned above, I doubt we will need to do this part in the exam) # Install a container runtime, IE https://github.com/containerd/containerd/blob/main/docs/getting-started.md sudo apt update && apt install containerd -y sudo apt-get update # apt-transport-https may be a dummy package; if so, you can skip that package sudo apt-get install -y apt-transport-https ca-certificates curl gpg # If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below. # sudo mkdir -p -m 755 /etc/apt/keyrings curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg # This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list sudo apt-get update sudo apt-get install -y kubelet kubeadm kubectl sudo apt-mark hold kubelet kubeadm kubectl You may need to execute the following depending on the underlying OS: modprobe br_netfilter echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables echo 1 > /proc/sys/net/ipv4/ip_forward Turn this node into a master sudo kubeadm init --pod-network-cidr=10.244.0.0/16 ... mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config ... kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml ... Note the join command, ie: kubeadm join 172.16.10.210:6443 --token 9tjntl.10plpxqy85g8a0ui \\ --discovery-token-ca-cert-hash sha256:381165c9a9f19a123bd0fee36fe36d15e918062dcc94711ff5b286ee1f86b92b Node 2: Run the join command taken from the previous step kubeadm join 172.16.10.210:6443 --token 9tjntl.10plpxqy85g8a0ui \\ --discovery-token-ca-cert-hash sha256:381165c9a9f19a123bd0fee36fe36d15e918062dcc94711ff5b286ee1f86b92b Validate by running kubectl get no on the master node: kubectl get no NAME STATUS ROLES AGE VERSION ubuntu Ready control-plane,master 9m53s v1.31.3 ubuntu2 Ready <none> 50s v1.31.3","title":"Exercise 2 - Use Kubeadm to install a basic cluster"},{"location":"lab-guide/01-architcture-installation-configuration/#exercise-3-manage-a-highly-available-kubernetes-cluster","text":"Using etcdctl , determine the health of the etcd cluster Using etcdctl , identify the list of members On the master node, determine the health of the cluster by probing the API endpoint Answer root@ip-172-31-31-80:~# ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \\ --cacert=/etc/kubernetes/pki/etcd/ca.crt \\ --cert=/etc/kubernetes/pki/etcd/server.crt \\ --key=/etc/kubernetes/pki/etcd/server.key \\ endpoint health https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 6.852757ms root@ip-172-31-31-80:~# ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \\ --cacert=/etc/kubernetes/pki/etcd/ca.crt \\ --cert=/etc/kubernetes/pki/etcd/server.crt \\ --key=/etc/kubernetes/pki/etcd/server.key \\ member list f90d942cc570125d, started, ip-172-31-31-80, https://172.31.31.80:2380, https://172.31.31.80:2379, false curl -k https://localhost:6443/healthz?verbose [+]ping ok [+]log ok [+]etcd ok [+]poststarthook/start-kube-apiserver-admission-initializer ok ...","title":"Exercise 3 - Manage a highly-available Kubernetes cluster"},{"location":"lab-guide/01-architcture-installation-configuration/#exercise-4-perform-a-version-upgrade-on-a-kubernetes-cluster-using-kubeadm","text":"Using kubeadm , upgrade a cluster to the lastest version Answer If held, unhold the kubeadm version sudo apt-mark unhold kubeadm Upgrade the kubeadm version: sudo apt-get install --only-upgrade kubeadm plan the upgrade: sudo kubeadm upgrade plan Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': COMPONENT CURRENT AVAILABLE kubelet 1 x v1.29.10 v1.31.3 Upgrade to the latest stable version: COMPONENT CURRENT AVAILABLE kube-apiserver v1.29.10 v1.31.3 kube-controller-manager v1.29.10 v1.31.3 kube-scheduler v1.29.10 v1.31.3 kube-proxy v1.29.10 v1.31.3 CoreDNS 1.7.0 1.11.3 etcd 3.4.9-1 3.5.15-0 Upgrade the cluster kubeadm upgrade apply v1.20.2 Upgrade Kubelet: sudo apt-get install --only-upgrade kubelet","title":"Exercise 4 - Perform a version upgrade on a Kubernetes cluster using Kubeadm"},{"location":"lab-guide/01-architcture-installation-configuration/#exercise-5-implement-etcd-backup-and-restore","text":"Take a backup of etcd Verify the etcd backup has been successful Restore the backup back to the cluster Answer Take a snapshot of etcd: ETCDCTL_API=3 etcdctl snapshot save snapshot.db --cacert /etc/kubernetes/pki/etcd/server.crt --cert /etc/kubernetes/pki/etcd/ca.crt --key /etc/kubernetes/pki/etcd/ca.key Verify the snapshot: sudo ETCDCTL_API=3 etcdctl --write-out=table snapshot status snapshot.db Perform a restore: ETCDCTL_API=3 etcdctl snapshot restore snapshot.db","title":"Exercise 5 - Implement etcd backup and restore"},{"location":"lab-guide/02-workloads-and-scheduling/","text":"Lab Exercises for Workloads & Scheduling Exercise 0 - Setup Prepare a cluster (Single node, kubeadm, k3s, etc) Open browser tabs to Kubernetes Documentation , Kubernetes GitHub and Kubernetes Blog (these are permitted as per the current guidelines ) Exercise 1 - Understand deployments and how to perform rolling update and rollbacks Create a deployment object consisting of 6 pods, each containing a single nginx:1.19.5 container Identify the update strategy adopted by this deployment Modify the update strategy so maxSurge is equal to 50% and maxUnavailable is equal to 50% Perform a rolling update to this deployment so that the image gets updated to nginx1.19.6 Undo the most recent change Answer - Imperative kubectl create deployment nginx --image=nginx:1.19.5 --replicas=6 kubectl describe deployment nginx | grep StrategyType kubectl patch deployment nginx -p \"{\\\"spec\\\": {\\\"strategy\\\": {\\\"rollingUpdate\\\": { \\\"maxSurge\\\":\\\"50%\\\"}}}}\" --record=true kubectl patch deployment nginx -p \"{\\\"spec\\\": {\\\"strategy\\\": {\\\"rollingUpdate\\\": { \\\"maxUnavailable\\\":\\\"50%\\\"}}}}\" --record=true kubectl set image deployment nginx nginx=nginx:1.19.6 --record=true kubectl rollout undo deployment/nginx Answer - Declarative apiVersion: apps/v1 kind: Deployment metadata: name: nginx labels: app: nginx spec: replicas: 6 strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.19.5 ports: - containerPort: 80 kubectl describe deployment nginx (unless specified, will be listed as \"UpdateStrategy\") patch.yaml contents: apiVersion: apps/v1 kind: Deployment metadata: name: nginx spec: strategy: rollingUpdate: maxSurge: 50% maxUnavailable: 50% template: spec: containers: - name: nginx image: nginx:1.19.6 kubectl patch deployment nginx --patch-file=patch.yaml kubectl rollout undo deployment/nginx Exercise 2 - Use ConfigMaps to configure applications Create a configmap named mycm that has the following key=value pair key = owner value = yourname Create a pod of your choice, such as nginx . Configure this Pod so that the underlying container has the environent varibale OWNER set to the value of this configmap Answer Create configmap: kubectl create configmap mycm --from-literal=owner=david Define Pod: apiVersion: v1 kind: Pod metadata: name: nginx-configmap spec: containers: - name: nginx-configmap image: nginx command: [ \"/bin/sh\", \"-c\", \"env\" ] env: # Define the environment variable - name: OWNER valueFrom: configMapKeyRef: # The ConfigMap containing the value you want to assign to SPECIAL_LEVEL_KEY name: mycm # Specify the key associated with the value key: owner Validate: kubectl logs nginx-configmap | grep OWNER OWNER=david Exercise 3 - Use Secrets to configure applications Create a secret named mysecret that has the following key=value pair dbuser = MyDatabaseUser dbpassword = MyDatabasePassword Create a pod of your choice, such as nginx . Configure this Pod so that the underlying container has the the following environment variables set: DBUSER from secret key dbuser DBPASS from secret key dbpassword Answer kubectl create secret generic mysecret --from-literal=dbuser=\"MyDatabaseUser\" --from-literal=dbpassword=\"MyDatabasePassword\" Apply the following manifest: apiVersion: v1 kind: Pod metadata: name: nginx-secret spec: containers: - name: nginx-secret image: nginx command: [ \"/bin/sh\", \"-c\", \"env\" ] env: - name: dbuser valueFrom: secretKeyRef: name: mysecret key: dbuser - name: dbpassword valueFrom: secretKeyRef: name: mysecret key: dbpassword restartPolicy: Never Which can be validated with: kubectl logs nginx-secret | grep db dbuser=MyDatabaseUser dbpassword=MyDatabasePassword Exercise 4 - Know how to scale applications Create a deployment object consisting of 3 pods containing a single nginx container Increase this deployment size by adding two additional pods. Decrease this deployment back to the original size of 3 pods Answer - Imperative kubectl create deployment nginx --image=nginx --replicas=3 kubectl scale --replicas=5 deployment nginx kubectl scale --replicas=3 deployment nginx Answer - Declarative Apply initial YAML: apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80 Apply modified YAML: apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: replicas: 5 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1 ports: - containerPort: 80 Apply original YAML: apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80 Excerise 5 - Understand how resource limits can affect Pod scheduling Create a new namespace called \"tenant-b-100mi\" Create a memory limit of 100Mi for this namespace Create a pod with a memory request of 150Mi, ensure the limit has been set by verifying you get a error message. Answer kubectl create ns tenant-b-100mi Deploy the manifest: apiVersion: v1 kind: LimitRange metadata: name: tenant-b-memlimit namespace: tenant-b-100mi spec: limits: - max: memory: 100Mi type: Container Test with deploying: apiVersion: v1 kind: Pod metadata: name: default-mem-demo namespace: tenant-b-100mi spec: containers: - name: default-mem-demo image: nginx resources: requests: memory: 150Mi Which should return: The Pod \"default-mem-demo\" is invalid: spec.containers[0].resources.requests: Invalid value: \"150Mi\": must be less than or equal to memory limit","title":"Part Two - Workloads & Scheduling"},{"location":"lab-guide/02-workloads-and-scheduling/#lab-exercises-for-workloads-scheduling","text":"","title":"Lab Exercises for Workloads &amp; Scheduling"},{"location":"lab-guide/02-workloads-and-scheduling/#exercise-0-setup","text":"Prepare a cluster (Single node, kubeadm, k3s, etc) Open browser tabs to Kubernetes Documentation , Kubernetes GitHub and Kubernetes Blog (these are permitted as per the current guidelines )","title":"Exercise 0 - Setup"},{"location":"lab-guide/02-workloads-and-scheduling/#exercise-1-understand-deployments-and-how-to-perform-rolling-update-and-rollbacks","text":"Create a deployment object consisting of 6 pods, each containing a single nginx:1.19.5 container Identify the update strategy adopted by this deployment Modify the update strategy so maxSurge is equal to 50% and maxUnavailable is equal to 50% Perform a rolling update to this deployment so that the image gets updated to nginx1.19.6 Undo the most recent change Answer - Imperative kubectl create deployment nginx --image=nginx:1.19.5 --replicas=6 kubectl describe deployment nginx | grep StrategyType kubectl patch deployment nginx -p \"{\\\"spec\\\": {\\\"strategy\\\": {\\\"rollingUpdate\\\": { \\\"maxSurge\\\":\\\"50%\\\"}}}}\" --record=true kubectl patch deployment nginx -p \"{\\\"spec\\\": {\\\"strategy\\\": {\\\"rollingUpdate\\\": { \\\"maxUnavailable\\\":\\\"50%\\\"}}}}\" --record=true kubectl set image deployment nginx nginx=nginx:1.19.6 --record=true kubectl rollout undo deployment/nginx Answer - Declarative apiVersion: apps/v1 kind: Deployment metadata: name: nginx labels: app: nginx spec: replicas: 6 strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.19.5 ports: - containerPort: 80 kubectl describe deployment nginx (unless specified, will be listed as \"UpdateStrategy\") patch.yaml contents: apiVersion: apps/v1 kind: Deployment metadata: name: nginx spec: strategy: rollingUpdate: maxSurge: 50% maxUnavailable: 50% template: spec: containers: - name: nginx image: nginx:1.19.6 kubectl patch deployment nginx --patch-file=patch.yaml kubectl rollout undo deployment/nginx","title":"Exercise 1 - Understand deployments and how to perform rolling update and rollbacks"},{"location":"lab-guide/02-workloads-and-scheduling/#exercise-2-use-configmaps-to-configure-applications","text":"Create a configmap named mycm that has the following key=value pair key = owner value = yourname Create a pod of your choice, such as nginx . Configure this Pod so that the underlying container has the environent varibale OWNER set to the value of this configmap Answer Create configmap: kubectl create configmap mycm --from-literal=owner=david Define Pod: apiVersion: v1 kind: Pod metadata: name: nginx-configmap spec: containers: - name: nginx-configmap image: nginx command: [ \"/bin/sh\", \"-c\", \"env\" ] env: # Define the environment variable - name: OWNER valueFrom: configMapKeyRef: # The ConfigMap containing the value you want to assign to SPECIAL_LEVEL_KEY name: mycm # Specify the key associated with the value key: owner Validate: kubectl logs nginx-configmap | grep OWNER OWNER=david","title":"Exercise 2 - Use ConfigMaps to configure applications"},{"location":"lab-guide/02-workloads-and-scheduling/#exercise-3-use-secrets-to-configure-applications","text":"Create a secret named mysecret that has the following key=value pair dbuser = MyDatabaseUser dbpassword = MyDatabasePassword Create a pod of your choice, such as nginx . Configure this Pod so that the underlying container has the the following environment variables set: DBUSER from secret key dbuser DBPASS from secret key dbpassword Answer kubectl create secret generic mysecret --from-literal=dbuser=\"MyDatabaseUser\" --from-literal=dbpassword=\"MyDatabasePassword\" Apply the following manifest: apiVersion: v1 kind: Pod metadata: name: nginx-secret spec: containers: - name: nginx-secret image: nginx command: [ \"/bin/sh\", \"-c\", \"env\" ] env: - name: dbuser valueFrom: secretKeyRef: name: mysecret key: dbuser - name: dbpassword valueFrom: secretKeyRef: name: mysecret key: dbpassword restartPolicy: Never Which can be validated with: kubectl logs nginx-secret | grep db dbuser=MyDatabaseUser dbpassword=MyDatabasePassword","title":"Exercise 3 - Use Secrets to configure applications"},{"location":"lab-guide/02-workloads-and-scheduling/#exercise-4-know-how-to-scale-applications","text":"Create a deployment object consisting of 3 pods containing a single nginx container Increase this deployment size by adding two additional pods. Decrease this deployment back to the original size of 3 pods Answer - Imperative kubectl create deployment nginx --image=nginx --replicas=3 kubectl scale --replicas=5 deployment nginx kubectl scale --replicas=3 deployment nginx Answer - Declarative Apply initial YAML: apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80 Apply modified YAML: apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: replicas: 5 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1 ports: - containerPort: 80 Apply original YAML: apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80","title":"Exercise 4 - Know how to scale applications"},{"location":"lab-guide/02-workloads-and-scheduling/#excerise-5-understand-how-resource-limits-can-affect-pod-scheduling","text":"Create a new namespace called \"tenant-b-100mi\" Create a memory limit of 100Mi for this namespace Create a pod with a memory request of 150Mi, ensure the limit has been set by verifying you get a error message. Answer kubectl create ns tenant-b-100mi Deploy the manifest: apiVersion: v1 kind: LimitRange metadata: name: tenant-b-memlimit namespace: tenant-b-100mi spec: limits: - max: memory: 100Mi type: Container Test with deploying: apiVersion: v1 kind: Pod metadata: name: default-mem-demo namespace: tenant-b-100mi spec: containers: - name: default-mem-demo image: nginx resources: requests: memory: 150Mi Which should return: The Pod \"default-mem-demo\" is invalid: spec.containers[0].resources.requests: Invalid value: \"150Mi\": must be less than or equal to memory limit","title":"Excerise 5 - Understand how resource limits can affect Pod scheduling"},{"location":"lab-guide/03-services-and-networking/","text":"Lab Exercises for Services & Networking Exercise 0 - Setup Prepare a cluster (Single node, kubeadm, k3s, etc) Open browser tabs to https://kubernetes.io/docs/ , https://github.com/kubernetes/ and https://kubernetes.io/blog/ (these are permitted as per the current guidelines ) Exercise 1 - Understand connectivity between Pods Deploy the following manifest Using kubectl , identify the Pod IP addresses Determine the DNS name of the service. Answer Identify the selector for the service: kubectl describe service nginx-service | grep -i selector Selector: app=nginx Filter kubectl output: kubectl get po -l app=nginx -o wide Service name will be, based on the format [Service Name].[Namespace].[Type].[Base Domain Name] : nginx-service.default.svc.cluster.local Exercise 2 - Understand ClusterIP, NodePort, LoadBalancer service types and endpoints Create three deployments of your choosing Expose one of these deployments with a service of type ClusterIP Expose one of these deployments with a service of type Nodeport Expose one of these deployments with a service of type Loadbalancer Note, this remains in pending status unless your cluster has integration with a cloud provider that provisions one for you (ie AWS ELB), or you have a software implementation such as metallb Answer - Imperative kubectl create deployment nginx-clusterip --image=nginx --replicas 1 kubectl create deployment nginx-nodeport --image=nginx --replicas 1 kubectl create deployment nginx-loadbalancer --image=nginx --replicas 1 kubectl expose deployment nginx-clusterip --type=\"ClusterIP\" --port=\"80\" kubectl expose deployment nginx-nodeport --type=\"NodePort\" --port=\"80\" kubectl expose deployment nginx-loadbalancer --type=\"LoadBalancer\" --port=\"80\" Answer - Declarative Apply the following: kind: Service apiVersion: v1 metadata: name: nginx-clusterip spec: selector: app: nginx-clusterip type: ClusterIP ports: - protocol: TCP port: 80 targetPort: 80 --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-clusterip labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx-clusterip template: metadata: labels: app: nginx-clusterip spec: containers: - name: nginx image: nginx ports: - containerPort: 80 --- kind: Service apiVersion: v1 metadata: name: nginx-nodeport spec: selector: app: nginx-nodeport type: NodePort ports: - protocol: TCP port: 80 targetPort: 80 --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-nodeport labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx-nodeport template: metadata: labels: app: nginx-nodeport spec: containers: - name: nginx image: nginx ports: - containerPort: 80 --- kind: Service apiVersion: v1 metadata: name: nginx-loadbalancer spec: selector: app: nginx-loadbalancer type: LoadBalancer ports: - port: 80 targetPort: 80 --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-loadbalancer labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx-loadbalancer template: metadata: labels: app: nginx-loadbalancer spec: containers: - name: nginx image: nginx ports: - containerPort: 80 Exercise 3 - Know how to use Ingress controllers and Ingress resources Create an ingress object named myingress with the following specification: Manages the host myingress.mydomain Traffic to the base path / will be forwarded to a service called main on port 80 Traffic to the path /api will be forwarded to a service called api on port 8080 Answer - Imperative kubectl create ingress myingress --rule=\"myingress.mydomain/=main:80\" --rule=\"myingress.mydomain/api=api:8080\" Answer - Declarative Apply the following YAML: apiVersion: networking.k8s.io/v1 kind: Ingress metadata: creationTimestamp: null name: myingress spec: rules: - host: myingress.mydomain http: paths: - backend: service: name: main port: number: 80 path: / pathType: Exact - backend: service: name: api port: number: 8080 path: /api pathType: Exact Exercise 4 - Know how to configure and use CoreDNS Identify the configuration location of coredns Modify the coredns config file so DNS queries not resolved by itself are forwarded to the DNS server 8.8.8.8 Validate the changes you have made Add additional configuration so that all DNS queries for custom.local are forwarded to the resolver 10.5.4.223 Answer kubectl get cm coredns -n kube-system NAME DATA AGE coredns 2 94d kubectl edit cm coredns -n kube-system replace: forward . /etc/resolv.conf with forward . 8.8.8.8 Add the block: custom.local:53 { errors cache 30 forward . 10.5.4.223 reload }","title":"Part Three - Services & Networking"},{"location":"lab-guide/03-services-and-networking/#lab-exercises-for-services-networking","text":"","title":"Lab Exercises for Services &amp; Networking"},{"location":"lab-guide/03-services-and-networking/#exercise-0-setup","text":"Prepare a cluster (Single node, kubeadm, k3s, etc) Open browser tabs to https://kubernetes.io/docs/ , https://github.com/kubernetes/ and https://kubernetes.io/blog/ (these are permitted as per the current guidelines )","title":"Exercise 0 - Setup"},{"location":"lab-guide/03-services-and-networking/#exercise-1-understand-connectivity-between-pods","text":"Deploy the following manifest Using kubectl , identify the Pod IP addresses Determine the DNS name of the service. Answer Identify the selector for the service: kubectl describe service nginx-service | grep -i selector Selector: app=nginx Filter kubectl output: kubectl get po -l app=nginx -o wide Service name will be, based on the format [Service Name].[Namespace].[Type].[Base Domain Name] : nginx-service.default.svc.cluster.local","title":"Exercise 1 - Understand connectivity between Pods"},{"location":"lab-guide/03-services-and-networking/#exercise-2-understand-clusterip-nodeport-loadbalancer-service-types-and-endpoints","text":"Create three deployments of your choosing Expose one of these deployments with a service of type ClusterIP Expose one of these deployments with a service of type Nodeport Expose one of these deployments with a service of type Loadbalancer Note, this remains in pending status unless your cluster has integration with a cloud provider that provisions one for you (ie AWS ELB), or you have a software implementation such as metallb Answer - Imperative kubectl create deployment nginx-clusterip --image=nginx --replicas 1 kubectl create deployment nginx-nodeport --image=nginx --replicas 1 kubectl create deployment nginx-loadbalancer --image=nginx --replicas 1 kubectl expose deployment nginx-clusterip --type=\"ClusterIP\" --port=\"80\" kubectl expose deployment nginx-nodeport --type=\"NodePort\" --port=\"80\" kubectl expose deployment nginx-loadbalancer --type=\"LoadBalancer\" --port=\"80\" Answer - Declarative Apply the following: kind: Service apiVersion: v1 metadata: name: nginx-clusterip spec: selector: app: nginx-clusterip type: ClusterIP ports: - protocol: TCP port: 80 targetPort: 80 --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-clusterip labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx-clusterip template: metadata: labels: app: nginx-clusterip spec: containers: - name: nginx image: nginx ports: - containerPort: 80 --- kind: Service apiVersion: v1 metadata: name: nginx-nodeport spec: selector: app: nginx-nodeport type: NodePort ports: - protocol: TCP port: 80 targetPort: 80 --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-nodeport labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx-nodeport template: metadata: labels: app: nginx-nodeport spec: containers: - name: nginx image: nginx ports: - containerPort: 80 --- kind: Service apiVersion: v1 metadata: name: nginx-loadbalancer spec: selector: app: nginx-loadbalancer type: LoadBalancer ports: - port: 80 targetPort: 80 --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-loadbalancer labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx-loadbalancer template: metadata: labels: app: nginx-loadbalancer spec: containers: - name: nginx image: nginx ports: - containerPort: 80","title":"Exercise 2 - Understand ClusterIP, NodePort, LoadBalancer service types and endpoints"},{"location":"lab-guide/03-services-and-networking/#exercise-3-know-how-to-use-ingress-controllers-and-ingress-resources","text":"Create an ingress object named myingress with the following specification: Manages the host myingress.mydomain Traffic to the base path / will be forwarded to a service called main on port 80 Traffic to the path /api will be forwarded to a service called api on port 8080 Answer - Imperative kubectl create ingress myingress --rule=\"myingress.mydomain/=main:80\" --rule=\"myingress.mydomain/api=api:8080\" Answer - Declarative Apply the following YAML: apiVersion: networking.k8s.io/v1 kind: Ingress metadata: creationTimestamp: null name: myingress spec: rules: - host: myingress.mydomain http: paths: - backend: service: name: main port: number: 80 path: / pathType: Exact - backend: service: name: api port: number: 8080 path: /api pathType: Exact","title":"Exercise 3 - Know how to use Ingress controllers and Ingress resources"},{"location":"lab-guide/03-services-and-networking/#exercise-4-know-how-to-configure-and-use-coredns","text":"Identify the configuration location of coredns Modify the coredns config file so DNS queries not resolved by itself are forwarded to the DNS server 8.8.8.8 Validate the changes you have made Add additional configuration so that all DNS queries for custom.local are forwarded to the resolver 10.5.4.223 Answer kubectl get cm coredns -n kube-system NAME DATA AGE coredns 2 94d kubectl edit cm coredns -n kube-system replace: forward . /etc/resolv.conf with forward . 8.8.8.8 Add the block: custom.local:53 { errors cache 30 forward . 10.5.4.223 reload }","title":"Exercise 4 - Know how to configure and use CoreDNS"},{"location":"lab-guide/04-storage/","text":"Lab Exercises for Storage Exercise 0 - Setup Prepare a cluster (Single node, kubeadm, k3s, etc) Open browser tabs to Kubernetes Documentation , Kubernetes GitHub and Kubernetes Blog (these are permitted as per the current guidelines ) Exercise 1 - Understand storage classes, persistent volumes Note: Leverage the docs to help you. It's unlikely you will need to recall the nuances from memory - Kubernetes Storage Classes Create a storageclass object named aws-ebs with the following specification: The provisioner should be kubernetes.io/aws-ebs The type is io1 The reclaimPolicy is Delete allowVolumeExpansion set to false Create a persistentvolume based on this storage class Note, it will not initialize unless AWS integration is configured, but the objective is to get the YAML correct. Answer Apply the following YAML to create the Storage Class apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: aws-ebs provisioner: kubernetes.io/aws-ebs parameters: type: io1 reclaimPolicy: Delete allowVolumeExpansion: false mountOptions: - debug volumeBindingMode: Immediate Apply the following YAML to create a Persistent Volume based on the Storage Class apiVersion: v1 kind: PersistentVolume metadata: name: pv-test spec: capacity: storage: 5Gi volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Recycle storageClassName: aws-ebs awsElasticBlockStore: fsType: \"ext4\" volumeID: \"vol-id\" validate with: kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pv-test 5Gi RWO Recycle Available aws-ebs 5 Exercise 2 - Know how to configure applications with persistent storage Create a persistentVolume object of type hostPath with the following parameters: 1GB Capacity Path on the host is /tmp storageClassName is Manual accessModes is ReadWriteOnce Create a persistentVolumeClaim to the aforementioned persistentVolume Create a pod workload to leverage this `persistentVolumeClaim To create the StorageClass manual apply the following: apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: manual provisioner: kubernetes.io/no-provisioner Answer apiVersion: v1 kind: PersistentVolume metadata: name: pv-hostpath-1gb spec: capacity: storage: 1Gi volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Recycle storageClassName: manual hostPath: path: /tmp --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-hostpath-claim spec: accessModes: - ReadWriteOnce volumeMode: Filesystem resources: requests: storage: 512Mi storageClassName: manual --- apiVersion: v1 kind: Pod metadata: name: pod-with-pvc spec: volumes: - name: myvol persistentVolumeClaim: claimName: pvc-hostpath-claim containers: - name: busybox image: busybox args: - sleep - \"1000000\" volumeMounts: - mountPath: \"/mnt/readonly\" name: myvol Validate with: kubectl get po,pv,pvc NAME READY STATUS RESTARTS AGE pod/pod-with-pvc 1/1 Running 0 2m7s NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pv-hostpath-1gb 1Gi RWO Recycle Bound default/pvc-hostpath-claim manual 2m7s NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/pvc-hostpath-claim Bound pv-hostpath-1gb 1Gi RWO manual 2m7s","title":"Part Four - Storage"},{"location":"lab-guide/04-storage/#lab-exercises-for-storage","text":"","title":"Lab Exercises for Storage"},{"location":"lab-guide/04-storage/#exercise-0-setup","text":"Prepare a cluster (Single node, kubeadm, k3s, etc) Open browser tabs to Kubernetes Documentation , Kubernetes GitHub and Kubernetes Blog (these are permitted as per the current guidelines )","title":"Exercise 0 - Setup"},{"location":"lab-guide/04-storage/#exercise-1-understand-storage-classes-persistent-volumes","text":"Note: Leverage the docs to help you. It's unlikely you will need to recall the nuances from memory - Kubernetes Storage Classes Create a storageclass object named aws-ebs with the following specification: The provisioner should be kubernetes.io/aws-ebs The type is io1 The reclaimPolicy is Delete allowVolumeExpansion set to false Create a persistentvolume based on this storage class Note, it will not initialize unless AWS integration is configured, but the objective is to get the YAML correct. Answer Apply the following YAML to create the Storage Class apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: aws-ebs provisioner: kubernetes.io/aws-ebs parameters: type: io1 reclaimPolicy: Delete allowVolumeExpansion: false mountOptions: - debug volumeBindingMode: Immediate Apply the following YAML to create a Persistent Volume based on the Storage Class apiVersion: v1 kind: PersistentVolume metadata: name: pv-test spec: capacity: storage: 5Gi volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Recycle storageClassName: aws-ebs awsElasticBlockStore: fsType: \"ext4\" volumeID: \"vol-id\" validate with: kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pv-test 5Gi RWO Recycle Available aws-ebs 5","title":"Exercise 1 - Understand storage classes, persistent volumes"},{"location":"lab-guide/04-storage/#exercise-2-know-how-to-configure-applications-with-persistent-storage","text":"Create a persistentVolume object of type hostPath with the following parameters: 1GB Capacity Path on the host is /tmp storageClassName is Manual accessModes is ReadWriteOnce Create a persistentVolumeClaim to the aforementioned persistentVolume Create a pod workload to leverage this `persistentVolumeClaim To create the StorageClass manual apply the following: apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: manual provisioner: kubernetes.io/no-provisioner Answer apiVersion: v1 kind: PersistentVolume metadata: name: pv-hostpath-1gb spec: capacity: storage: 1Gi volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Recycle storageClassName: manual hostPath: path: /tmp --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-hostpath-claim spec: accessModes: - ReadWriteOnce volumeMode: Filesystem resources: requests: storage: 512Mi storageClassName: manual --- apiVersion: v1 kind: Pod metadata: name: pod-with-pvc spec: volumes: - name: myvol persistentVolumeClaim: claimName: pvc-hostpath-claim containers: - name: busybox image: busybox args: - sleep - \"1000000\" volumeMounts: - mountPath: \"/mnt/readonly\" name: myvol Validate with: kubectl get po,pv,pvc NAME READY STATUS RESTARTS AGE pod/pod-with-pvc 1/1 Running 0 2m7s NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pv-hostpath-1gb 1Gi RWO Recycle Bound default/pvc-hostpath-claim manual 2m7s NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/pvc-hostpath-claim Bound pv-hostpath-1gb 1Gi RWO manual 2m7s","title":"Exercise 2 - Know how to configure applications with persistent storage"},{"location":"lab-guide/05-troubleshooting/","text":"Lab Exercises for Troubleshooting Exercise 0 - Setup Prepare a cluster (Single node, kubeadm, k3s, etc) Open browser tabs to Kubernetes Documentation , Kubernetes GitHub and Kubernetes Blog (these are permitted as per the current guidelines ) Ensure your etcd nodes have etcdctl installed Exercise 1 - Evaluate cluster and node logging For your cluster type, determine how to acquire logs for your master nodes. They could be in the form of: Services Static Pods Kubernetes Pods Using kubectl get a list of events from your cluster Using etcdctl determine the health of the etcd cluster Answer 1 Is dependent on how the cluster was made and potentially which OS's were used. For example, if K8S components manifest as Kubernetes Pods: kubectl logs <podname> <namespace> kubectl get events etcdctl --write-out=table --endpoints=$ENDPOINTS endpoint status +------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | https://127.0.0.1:2379 | 4e30a295f2c3c1a4 | 3.5.0 | 8.1 MB | true | false | 3 | 7903 | 7903 | | +------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ Exercise 2 - Understand how to monitor applications Deploy the following manifest: counter-pod.yaml Acquire the logs from this pod - what do they mention? Answer kubectl logs counter 0: Mon Dec 9 13:15:08 UTC 2024 1: Mon Dec 9 13:15:09 UTC 2024 2: Mon Dec 9 13:15:10 UTC 2024 3: Mon Dec 9 13:15:11 UTC 2024 4: Mon Dec 9 13:15:12 UTC 2024 5: Mon Dec 9 13:15:13 UTC 2024 6: Mon Dec 9 13:15:14 UTC 2024 7: Mon Dec 9 13:15:15 UTC 2024 8: Mon Dec 9 13:15:16 UTC 2024 9: Mon Dec 9 13:15:17 UTC 2024 Exercise 3 - Troubleshoot application failure Deploy the following manifest: brokenpod.yaml . It will create a Pod named nginx-pod . Determine why this Pod will not enter running state by using kubectl Answer kubectl describe pod nginx-pod .. Normal BackOff 4m55s (x7 over 6m56s) kubelet Back-off pulling image \"nginx:invalidversion\" .. Exercise 4 - Troubleshoot networking Deploy the following manifest: https://raw.githubusercontent.com/David-VTUK/CKAExampleYaml/master/nginx-svc-and-deployment-broken.yaml , It will create deployment and service objects. Identify the DNS name of this service. Test resolution of this DNS record: Create a Pod that has nslookup installed. ie: kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml Test sending traffic to this service Create a Pod that has curl installed. ie: kubectl run curl --image=radial/busyboxplus:curl -i --tty - Why does it fail? Rectify the identified error in step 3 Answer nginx-service.default.svc.cluster.local kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml kubectl exec -it dnsutils sh / # nslookup nginx-service.default.svc.cluster.local Server: 10.96.0.10 Address: 10.96.0.10#53 Name: nginx-service.default.svc.cluster.local Address: 10.99.41.254 kubectl run curl --image=radial/busyboxplus:curl -i --tty If you don't see a command prompt, try pressing enter. [ root@curl:/ ]$ curl nginx-service.default.svc.cluster.local curl: (7) Failed to connect to nginx-service.default.svc.cluster.local port 80: Connection refused Check service: kubectl describe service nginx-service Name: nginx-service Namespace: default Labels: <none> Annotations: <none> Selector: app=nginx Type: ClusterIP IP Families: <none> IP: 10.99.41.254 IPs: 10.99.41.254 Port: <unset> 80/TCP TargetPort: 8080/TCP Endpoints: 10.244.1.18:8080,10.244.1.19:8080,10.244.1.20:8080 Session Affinity: None Events: <none> Note: Service is listening on port 80 Service has a endpoint list, with target port of 8080 Test curl directly against pod: curl 10.244.1.18:8080 curl: (7) Failed to connect to 10.244.1.18 port 8080: Connection refused Port 8080 isn't listening, check the pod config: kubectl describe po nginx-deployment-5d59d67564-bk9xb | grep -i \"port:\" Port: 80/TCP The service is trying to forward traffic to port 8080 on the container, but the container is only listening on port 80. Reconfigure the service object, ie: kubectl edit service nginx-service Replace targetPort: 8080 with targetPort: 80 Retest: curl nginx-service.default.svc.cluster.local <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title>","title":"Part Five - Troubleshooting"},{"location":"lab-guide/05-troubleshooting/#lab-exercises-for-troubleshooting","text":"","title":"Lab Exercises for Troubleshooting"},{"location":"lab-guide/05-troubleshooting/#exercise-0-setup","text":"Prepare a cluster (Single node, kubeadm, k3s, etc) Open browser tabs to Kubernetes Documentation , Kubernetes GitHub and Kubernetes Blog (these are permitted as per the current guidelines ) Ensure your etcd nodes have etcdctl installed","title":"Exercise 0 - Setup"},{"location":"lab-guide/05-troubleshooting/#exercise-1-evaluate-cluster-and-node-logging","text":"For your cluster type, determine how to acquire logs for your master nodes. They could be in the form of: Services Static Pods Kubernetes Pods Using kubectl get a list of events from your cluster Using etcdctl determine the health of the etcd cluster Answer 1 Is dependent on how the cluster was made and potentially which OS's were used. For example, if K8S components manifest as Kubernetes Pods: kubectl logs <podname> <namespace> kubectl get events etcdctl --write-out=table --endpoints=$ENDPOINTS endpoint status +------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | https://127.0.0.1:2379 | 4e30a295f2c3c1a4 | 3.5.0 | 8.1 MB | true | false | 3 | 7903 | 7903 | | +------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+","title":"Exercise 1 - Evaluate cluster and node logging"},{"location":"lab-guide/05-troubleshooting/#exercise-2-understand-how-to-monitor-applications","text":"Deploy the following manifest: counter-pod.yaml Acquire the logs from this pod - what do they mention? Answer kubectl logs counter 0: Mon Dec 9 13:15:08 UTC 2024 1: Mon Dec 9 13:15:09 UTC 2024 2: Mon Dec 9 13:15:10 UTC 2024 3: Mon Dec 9 13:15:11 UTC 2024 4: Mon Dec 9 13:15:12 UTC 2024 5: Mon Dec 9 13:15:13 UTC 2024 6: Mon Dec 9 13:15:14 UTC 2024 7: Mon Dec 9 13:15:15 UTC 2024 8: Mon Dec 9 13:15:16 UTC 2024 9: Mon Dec 9 13:15:17 UTC 2024","title":"Exercise 2 - Understand how to monitor applications"},{"location":"lab-guide/05-troubleshooting/#exercise-3-troubleshoot-application-failure","text":"Deploy the following manifest: brokenpod.yaml . It will create a Pod named nginx-pod . Determine why this Pod will not enter running state by using kubectl Answer kubectl describe pod nginx-pod .. Normal BackOff 4m55s (x7 over 6m56s) kubelet Back-off pulling image \"nginx:invalidversion\" ..","title":"Exercise 3 - Troubleshoot application failure"},{"location":"lab-guide/05-troubleshooting/#exercise-4-troubleshoot-networking","text":"Deploy the following manifest: https://raw.githubusercontent.com/David-VTUK/CKAExampleYaml/master/nginx-svc-and-deployment-broken.yaml , It will create deployment and service objects. Identify the DNS name of this service. Test resolution of this DNS record: Create a Pod that has nslookup installed. ie: kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml Test sending traffic to this service Create a Pod that has curl installed. ie: kubectl run curl --image=radial/busyboxplus:curl -i --tty - Why does it fail? Rectify the identified error in step 3 Answer nginx-service.default.svc.cluster.local kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml kubectl exec -it dnsutils sh / # nslookup nginx-service.default.svc.cluster.local Server: 10.96.0.10 Address: 10.96.0.10#53 Name: nginx-service.default.svc.cluster.local Address: 10.99.41.254 kubectl run curl --image=radial/busyboxplus:curl -i --tty If you don't see a command prompt, try pressing enter. [ root@curl:/ ]$ curl nginx-service.default.svc.cluster.local curl: (7) Failed to connect to nginx-service.default.svc.cluster.local port 80: Connection refused Check service: kubectl describe service nginx-service Name: nginx-service Namespace: default Labels: <none> Annotations: <none> Selector: app=nginx Type: ClusterIP IP Families: <none> IP: 10.99.41.254 IPs: 10.99.41.254 Port: <unset> 80/TCP TargetPort: 8080/TCP Endpoints: 10.244.1.18:8080,10.244.1.19:8080,10.244.1.20:8080 Session Affinity: None Events: <none> Note: Service is listening on port 80 Service has a endpoint list, with target port of 8080 Test curl directly against pod: curl 10.244.1.18:8080 curl: (7) Failed to connect to 10.244.1.18 port 8080: Connection refused Port 8080 isn't listening, check the pod config: kubectl describe po nginx-deployment-5d59d67564-bk9xb | grep -i \"port:\" Port: 80/TCP The service is trying to forward traffic to port 8080 on the container, but the container is only listening on port 80. Reconfigure the service object, ie: kubectl edit service nginx-service Replace targetPort: 8080 with targetPort: 80 Retest: curl nginx-service.default.svc.cluster.local <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title>","title":"Exercise 4 - Troubleshoot networking"},{"location":"revision-topics/01-architcture-installation-configuration/","text":"Cluster Architecture, Installation & Configuration Manage Role Based Access Control (RBAC) Kubernetes implements an RBAC framework to govern access to resources within a cluster and forms part of the overall Authentication, Authorization and Admission control framework. To determine who (or what) has access to which resources, a number of steps have to be executed. Step 1 - Authentication First step is Authentication which is how a user or service account identifies itself. Depending on the source, a corresponding authentication module is used. Authentication modules include the ability to authenticate from the following: Client Certificate Password Plain Tokens Bootstrap Tokens JWT Tokens (for service accounts) All authentication is handled via HTTP over TLS. Step 2 - Authorization After a user or service account is authenticated, the request must then be authorized. Any authentication request is followed by some kind of action request, and the action defines the object(s) that request needs to apply to, and what the action is. For example, to list the pods in a given namespace. Any and all requests are facilitated providing an existing policy gives the user those permissions. Steps 3 & 4 - Admission Control Admission Control Modules are software modules that can modify or reject requests. In addition to the attributes available to Authorization Modules, Admission Control Modules can access the contents of the object that is being created or updated. They act on objects being created, deleted, updated or connected (proxy), but not reads. Role and Rolebindings Implementing RBAC rules largely involves two object types within Kubernetes - role and rolebindings : A role grants access to resources within a single namespace. A rolebinding grants the permissions from a role to a user, group or service account within a single namespace. clusterrole and clusterrolebindings operate similarly, but obviously provide access to non-namespaced resources. kubectl api-resources --namespaced=false can be used to determine which resource types are not namespaced. Examples include: node , persistentvolume , storageclass and users . Users can either be serviceaccounts or users . The former is typically used to authenticate applications, the latter for human users. To test, the below creates namespace , serviceaccount , role and rolebinding apiVersion: v1 kind: Namespace metadata: name: rbac-test apiVersion: v1 kind: ServiceAccount metadata: name: rbac-test-sa namespace: rbac-test ``` of particular importance is the format of the below. `apiGroup` : Determines which API group to apply this to. `resources`: Which resource types to apply this to. `verbs`: What we can do to these objects (ie create, delete, watch, etc) ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: rbac-test-role namespace: rbac-test rules: - apiGroups: [\"\"] resources: [\"pods\"] verbs: [\"get\", \"list\", \"watch\"] apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: rbac-test-rolebinding namespace: rbac-test roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: rbac-test-role subjects: - kind: ServiceAccount name: rbac-test-sa namespace: rbac-test We can then validate this with kubectl. The following returns yes as that service account can get pods kubectl -n rbac-test --as=system:serviceaccount:rbac-test:rbac-test-sa auth can-i get pods yes However with secrets , it returns no kubectl -n rbac-test --as=system:serviceaccount:rbac-test:rbac-test-sa auth can-i get secrets no Use Kubeadm to install a basic cluster kubeadm is a utility to bootstrap Kubernetes to a number of existing, vanilla nodes. It takes care of the etcd cluster, Kubernetes master and worker nodes including all the required components to instantiate a viable minimum k8s cluster. What you get at the end of using kubeadm is a fully working, fully functioning kubernetes cluster. It's at the opposite end of the spectrum in terms of difficulty compared to, for example, Kelsey Hightower's \"Kubernetes the hard way\". For the exam, it is recommended that you become familiar with both ways of deploying Kubernetes clusters. Kubeadm is a command line utility that performs the following functions: kubeadm init to bootstrap a Kubernetes control-plane node kubeadm join to bootstrap a Kubernetes worker node and join it to the cluster kubeadm upgrade to upgrade a Kubernetes cluster to a newer version kubeadm config if you initialized your cluster using kubeadm v1.7.x or lower, to configure your cluster for kubeadm upgrade kubeadm token to manage tokens for kubeadm join kubeadm reset to revert any changes made to this host by kubeadm init or kubeadm join kubeadm version to print the kubeadm version kubeadm alpha to preview a set of features made available for gathering feedback from the community Kubeadm - Master Node Install In the following examples 3x Ubuntu Server VMs were created k8s-cl02-ms01 k8s-cl02-wk01 k8s-cl02-wk02 Where appropriate, ensure your nodes have a container runtime installed. On the master node install the required binaries apt-get update && apt-get install -y apt-transport-https curl curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - cat <<EOF >/etc/apt/sources.list.d/kubernetes.list deb https://apt.kubernetes.io/ kubernetes-xenial main EOF apt-get update apt-get install -y kubelet kubeadm kubectl apt-mark hold kubelet kubeadm kubectl Initialise the master node: sudo kubeadm init --pod-network-cidr=10.244.0.0/16 Note, the requirement to pass --pod-network is dependent on the chosen CNI. For Flannel, this is required. Kubeadm will also let you know if any prerequisites are not made. Once completed, a message will be displayed: Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run \"kubectl apply -f [podnetwork].yaml\" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 172.16.10.80:6443 --token j5nqhd.cnfmnjgc68aato60 \\ --discovery-token-ca-cert-hash sha256:cbc91031c1ffa47bbea83aa1cf65e99821a1f582c4363e1a4408715bfd66bb60 Some important pieces of information to note: Kubeadm has created the admin kubeconfig file for you, and recommends copying this to the logged on users home directory for ease Kubeadm has not deployed a pod networking solution yet. Therefore, this is a post-install activity Kubeadm has provided a join command together with a token to add worker nodes. We can regenerate this token if required. If we issue a kubectl get nodes command we will see the master node is not ready NAME STATUS ROLES AGE VERSION k8s-cl02-ms01 NotReady master 6m20s v1.20.2 As per the output of kubeadm, install a network solution, Ie flannel. For flannel to work correctly, you must pass --pod-network-cidr=10.244.0.0/16 to kubeadm init. Additionally, set /proc/sys/net/bridge/bridge-nf-call-iptables to 1 by running sysctl net.bridge.bridge-nf-call-iptables=1 . Install Flannel: kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml After a few seconds, the master node will now be ready NAME STATUS ROLES AGE VERSION k8s-cl02-ms01 Ready master 10m v1.20.2 Kubeadm - Install worker nodes The installation process for worker nodes is similar to master nodes - the only exception is we do not execute the \"kubeadm init\" command, as this is only run on masters. For workers, we use \"kubeadm join\". As prep: Install a container runtime Install the kubeadm binaries (as above) To join a worker node to a cluster created by kubeadm we need to use the kubeadm join command with a token generated on the master. This is shown after we run kubeadm init on the master node. However, we can easily regenerate this on the master node should it not be noted down or expired: (on the master node) david@k8s-cl02-ms01:~$ kubeadm token create --print-join-command kubeadm join 172.16.10.80:6443 --token ht55yv.8lq69q0189xhe2ql --discovery-token-ca-cert-hash sha256:cbc91031c1ffa47bbea83aa1cf65e99821a1f582c4363e1a4408715bfd66bb60 Use this command on the worker (as root) root@k8s-cl02-wk01:~# kubeadm join 172.16.10.80:6443 --token ht55yv.8lq69q0189xhe2ql --discovery-token-ca-cert-hash sha256:cbc91031c1ffa47bbea83aa1cf65e99821a1f582c4363e1a4408715bfd66bb60 After which confirmation will be displayed: This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. To validate, run kubectl get nodes on the master node: NAME STATUS ROLES AGE VERSION k8s-cl02-ms01 Ready master 50m v1.20.2 k8s-cl02-wk01 Ready <none> 2m10s v1.20.2 Manage a highly-available Kubernetes cluster The previous section demonstrated creating a K8s cluster with one master node and several worker nodes - this does not provide resilience for the control plane. Several topologies exist for doing so: Stacked etcd Multiple worker nodes Multiple control plane nodes fronted by a loadbalancer Embedded etcd within control plane Notes: etcd is quorum based. Therefore, if using stacked control plane nodes with etcd, odd numbers must be used. External etcd Notes: Multiple worker nodes Multiple control plane nodes fronted by a loadbalancer Etcd is external of the k8s cluster Notes: Advantage with this setup is etcd and the control plane can be scaled and managed independently of each other. This provides greater flexibility at the expense of operational complexity. Assessing cluster health kubectl get componentstatus is deprecated as of 1.20. A suitable replacement includes probing the API server directly, For example, on a master node, run curl -k https://localhost:6443/livez?verbose which returns: [+]ping ok [+]log ok [+]etcd ok [+]poststarthook/start-kube-apiserver-admission-initializer ok [+]poststarthook/generic-apiserver-start-informers ok .....etc Three endpoints exist - healthz , livez and readyz to indicate the current status of the API server Provision underlying infrastructure to deploy a Kubernetes cluster The topology choices above will influence the underlying resources that need to be provisioned. How these are provisioned are specific to the underlying cloud provider. Some generic observations: Disable swap. Leverage cloud capabilities for HA - ie using multiple AZ's. Windows can be used for worker nodes, but not control plane. Perform a version upgrade on a Kubernetes cluster using Kubeadm First, install kubeadm to a specific version. This will determine the k8s version that it deploys: sudo apt-get update && sudo apt-get install -y kubeadm=1.19.0-00 kubelet=1.19.0-00 kubectl=1.19.0-00 && sudo apt-mark hold kubeadm Stand up a k8s cluster sudo kubeadm init --pod-network-cidr=10.244.0.0/16 Add CNI https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml To upgrade the underlying k8s cluster, we need to upgrade kubeadm. Update kubeadm sudo apt-mark unhold kubeadm sudo apt-get install --only-upgrade kubeadm Next we plan the upgrade - this won't change our cluster but will display what changes can be made: sudo kubeadm upgrade plan Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': COMPONENT CURRENT AVAILABLE kubelet 1 x v1.19.0 v1.20.2 Upgrade to the latest stable version: COMPONENT CURRENT AVAILABLE kube-apiserver v1.19.7 v1.20.2 kube-controller-manager v1.19.7 v1.20.2 kube-scheduler v1.19.7 v1.20.2 kube-proxy v1.19.7 v1.20.2 CoreDNS 1.7.0 1.7.0 etcd 3.4.9-1 3.4.13-0 You can now apply the upgrade by executing the following command: kubeadm upgrade apply v1.20.2 Important note: kubelet must be upgraded manually after this step. Upgrade the cluster: kubeadm upgrade apply v1.20.2 upgrade Kubelet: sudo apt-get install --only-upgrade kubelet kubectl Implement etcd backup and restore Backing up etcd Take a snapshot of the DB, then store it in a safe location: ETCDCTL_API=3 etcdctl snapshot save snapshot.db --cacert /etc/kubernetes/pki/etcd/server.crt --cert /etc/kubernetes/pki/etcd/ca.crt --key /etc/kubernetes/pki/etcd/ca.key Verify the backup: sudo ETCDCTL_API=3 etcdctl --write-out=table snapshot status snapshot.db +----------+----------+------------+------------+ | HASH | REVISION | TOTAL KEYS | TOTAL SIZE | +----------+----------+------------+------------+ | 2125d542 | 364069 | 770 | 3.8 MB | +----------+----------+------------+------------+ Restore to etcd To perform a restore: ETCDCTL_API=3 etcdctl snapshot restore snapshot.db","title":"Part One - Architecture, Installation and Configuration"},{"location":"revision-topics/01-architcture-installation-configuration/#cluster-architecture-installation-configuration","text":"","title":"Cluster Architecture, Installation &amp; Configuration"},{"location":"revision-topics/01-architcture-installation-configuration/#manage-role-based-access-control-rbac","text":"Kubernetes implements an RBAC framework to govern access to resources within a cluster and forms part of the overall Authentication, Authorization and Admission control framework. To determine who (or what) has access to which resources, a number of steps have to be executed.","title":"Manage Role Based Access Control (RBAC)"},{"location":"revision-topics/01-architcture-installation-configuration/#step-1-authentication","text":"First step is Authentication which is how a user or service account identifies itself. Depending on the source, a corresponding authentication module is used. Authentication modules include the ability to authenticate from the following: Client Certificate Password Plain Tokens Bootstrap Tokens JWT Tokens (for service accounts) All authentication is handled via HTTP over TLS.","title":"Step 1 - Authentication"},{"location":"revision-topics/01-architcture-installation-configuration/#step-2-authorization","text":"After a user or service account is authenticated, the request must then be authorized. Any authentication request is followed by some kind of action request, and the action defines the object(s) that request needs to apply to, and what the action is. For example, to list the pods in a given namespace. Any and all requests are facilitated providing an existing policy gives the user those permissions.","title":"Step 2 - Authorization"},{"location":"revision-topics/01-architcture-installation-configuration/#steps-3-4-admission-control","text":"Admission Control Modules are software modules that can modify or reject requests. In addition to the attributes available to Authorization Modules, Admission Control Modules can access the contents of the object that is being created or updated. They act on objects being created, deleted, updated or connected (proxy), but not reads.","title":"Steps 3 &amp; 4 - Admission Control"},{"location":"revision-topics/01-architcture-installation-configuration/#role-and-rolebindings","text":"Implementing RBAC rules largely involves two object types within Kubernetes - role and rolebindings : A role grants access to resources within a single namespace. A rolebinding grants the permissions from a role to a user, group or service account within a single namespace. clusterrole and clusterrolebindings operate similarly, but obviously provide access to non-namespaced resources. kubectl api-resources --namespaced=false can be used to determine which resource types are not namespaced. Examples include: node , persistentvolume , storageclass and users . Users can either be serviceaccounts or users . The former is typically used to authenticate applications, the latter for human users. To test, the below creates namespace , serviceaccount , role and rolebinding apiVersion: v1 kind: Namespace metadata: name: rbac-test apiVersion: v1 kind: ServiceAccount metadata: name: rbac-test-sa namespace: rbac-test ``` of particular importance is the format of the below. `apiGroup` : Determines which API group to apply this to. `resources`: Which resource types to apply this to. `verbs`: What we can do to these objects (ie create, delete, watch, etc) ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: rbac-test-role namespace: rbac-test rules: - apiGroups: [\"\"] resources: [\"pods\"] verbs: [\"get\", \"list\", \"watch\"] apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: rbac-test-rolebinding namespace: rbac-test roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: rbac-test-role subjects: - kind: ServiceAccount name: rbac-test-sa namespace: rbac-test We can then validate this with kubectl. The following returns yes as that service account can get pods kubectl -n rbac-test --as=system:serviceaccount:rbac-test:rbac-test-sa auth can-i get pods yes However with secrets , it returns no kubectl -n rbac-test --as=system:serviceaccount:rbac-test:rbac-test-sa auth can-i get secrets no","title":"Role and Rolebindings"},{"location":"revision-topics/01-architcture-installation-configuration/#use-kubeadm-to-install-a-basic-cluster","text":"kubeadm is a utility to bootstrap Kubernetes to a number of existing, vanilla nodes. It takes care of the etcd cluster, Kubernetes master and worker nodes including all the required components to instantiate a viable minimum k8s cluster. What you get at the end of using kubeadm is a fully working, fully functioning kubernetes cluster. It's at the opposite end of the spectrum in terms of difficulty compared to, for example, Kelsey Hightower's \"Kubernetes the hard way\". For the exam, it is recommended that you become familiar with both ways of deploying Kubernetes clusters. Kubeadm is a command line utility that performs the following functions: kubeadm init to bootstrap a Kubernetes control-plane node kubeadm join to bootstrap a Kubernetes worker node and join it to the cluster kubeadm upgrade to upgrade a Kubernetes cluster to a newer version kubeadm config if you initialized your cluster using kubeadm v1.7.x or lower, to configure your cluster for kubeadm upgrade kubeadm token to manage tokens for kubeadm join kubeadm reset to revert any changes made to this host by kubeadm init or kubeadm join kubeadm version to print the kubeadm version kubeadm alpha to preview a set of features made available for gathering feedback from the community","title":"Use Kubeadm to install a basic cluster"},{"location":"revision-topics/01-architcture-installation-configuration/#kubeadm-master-node-install","text":"In the following examples 3x Ubuntu Server VMs were created k8s-cl02-ms01 k8s-cl02-wk01 k8s-cl02-wk02 Where appropriate, ensure your nodes have a container runtime installed. On the master node install the required binaries apt-get update && apt-get install -y apt-transport-https curl curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - cat <<EOF >/etc/apt/sources.list.d/kubernetes.list deb https://apt.kubernetes.io/ kubernetes-xenial main EOF apt-get update apt-get install -y kubelet kubeadm kubectl apt-mark hold kubelet kubeadm kubectl Initialise the master node: sudo kubeadm init --pod-network-cidr=10.244.0.0/16 Note, the requirement to pass --pod-network is dependent on the chosen CNI. For Flannel, this is required. Kubeadm will also let you know if any prerequisites are not made. Once completed, a message will be displayed: Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run \"kubectl apply -f [podnetwork].yaml\" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 172.16.10.80:6443 --token j5nqhd.cnfmnjgc68aato60 \\ --discovery-token-ca-cert-hash sha256:cbc91031c1ffa47bbea83aa1cf65e99821a1f582c4363e1a4408715bfd66bb60 Some important pieces of information to note: Kubeadm has created the admin kubeconfig file for you, and recommends copying this to the logged on users home directory for ease Kubeadm has not deployed a pod networking solution yet. Therefore, this is a post-install activity Kubeadm has provided a join command together with a token to add worker nodes. We can regenerate this token if required. If we issue a kubectl get nodes command we will see the master node is not ready NAME STATUS ROLES AGE VERSION k8s-cl02-ms01 NotReady master 6m20s v1.20.2 As per the output of kubeadm, install a network solution, Ie flannel. For flannel to work correctly, you must pass --pod-network-cidr=10.244.0.0/16 to kubeadm init. Additionally, set /proc/sys/net/bridge/bridge-nf-call-iptables to 1 by running sysctl net.bridge.bridge-nf-call-iptables=1 . Install Flannel: kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml After a few seconds, the master node will now be ready NAME STATUS ROLES AGE VERSION k8s-cl02-ms01 Ready master 10m v1.20.2","title":"Kubeadm - Master Node Install"},{"location":"revision-topics/01-architcture-installation-configuration/#kubeadm-install-worker-nodes","text":"The installation process for worker nodes is similar to master nodes - the only exception is we do not execute the \"kubeadm init\" command, as this is only run on masters. For workers, we use \"kubeadm join\". As prep: Install a container runtime Install the kubeadm binaries (as above) To join a worker node to a cluster created by kubeadm we need to use the kubeadm join command with a token generated on the master. This is shown after we run kubeadm init on the master node. However, we can easily regenerate this on the master node should it not be noted down or expired: (on the master node) david@k8s-cl02-ms01:~$ kubeadm token create --print-join-command kubeadm join 172.16.10.80:6443 --token ht55yv.8lq69q0189xhe2ql --discovery-token-ca-cert-hash sha256:cbc91031c1ffa47bbea83aa1cf65e99821a1f582c4363e1a4408715bfd66bb60 Use this command on the worker (as root) root@k8s-cl02-wk01:~# kubeadm join 172.16.10.80:6443 --token ht55yv.8lq69q0189xhe2ql --discovery-token-ca-cert-hash sha256:cbc91031c1ffa47bbea83aa1cf65e99821a1f582c4363e1a4408715bfd66bb60 After which confirmation will be displayed: This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. To validate, run kubectl get nodes on the master node: NAME STATUS ROLES AGE VERSION k8s-cl02-ms01 Ready master 50m v1.20.2 k8s-cl02-wk01 Ready <none> 2m10s v1.20.2","title":"Kubeadm - Install worker nodes"},{"location":"revision-topics/01-architcture-installation-configuration/#manage-a-highly-available-kubernetes-cluster","text":"The previous section demonstrated creating a K8s cluster with one master node and several worker nodes - this does not provide resilience for the control plane. Several topologies exist for doing so:","title":"Manage a highly-available Kubernetes cluster"},{"location":"revision-topics/01-architcture-installation-configuration/#stacked-etcd","text":"Multiple worker nodes Multiple control plane nodes fronted by a loadbalancer Embedded etcd within control plane Notes: etcd is quorum based. Therefore, if using stacked control plane nodes with etcd, odd numbers must be used.","title":"Stacked etcd"},{"location":"revision-topics/01-architcture-installation-configuration/#external-etcd","text":"Notes: Multiple worker nodes Multiple control plane nodes fronted by a loadbalancer Etcd is external of the k8s cluster Notes: Advantage with this setup is etcd and the control plane can be scaled and managed independently of each other. This provides greater flexibility at the expense of operational complexity.","title":"External etcd"},{"location":"revision-topics/01-architcture-installation-configuration/#assessing-cluster-health","text":"kubectl get componentstatus is deprecated as of 1.20. A suitable replacement includes probing the API server directly, For example, on a master node, run curl -k https://localhost:6443/livez?verbose which returns: [+]ping ok [+]log ok [+]etcd ok [+]poststarthook/start-kube-apiserver-admission-initializer ok [+]poststarthook/generic-apiserver-start-informers ok .....etc Three endpoints exist - healthz , livez and readyz to indicate the current status of the API server","title":"Assessing cluster health"},{"location":"revision-topics/01-architcture-installation-configuration/#provision-underlying-infrastructure-to-deploy-a-kubernetes-cluster","text":"The topology choices above will influence the underlying resources that need to be provisioned. How these are provisioned are specific to the underlying cloud provider. Some generic observations: Disable swap. Leverage cloud capabilities for HA - ie using multiple AZ's. Windows can be used for worker nodes, but not control plane.","title":"Provision underlying infrastructure to deploy a Kubernetes cluster"},{"location":"revision-topics/01-architcture-installation-configuration/#perform-a-version-upgrade-on-a-kubernetes-cluster-using-kubeadm","text":"First, install kubeadm to a specific version. This will determine the k8s version that it deploys: sudo apt-get update && sudo apt-get install -y kubeadm=1.19.0-00 kubelet=1.19.0-00 kubectl=1.19.0-00 && sudo apt-mark hold kubeadm Stand up a k8s cluster sudo kubeadm init --pod-network-cidr=10.244.0.0/16 Add CNI https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml To upgrade the underlying k8s cluster, we need to upgrade kubeadm. Update kubeadm sudo apt-mark unhold kubeadm sudo apt-get install --only-upgrade kubeadm Next we plan the upgrade - this won't change our cluster but will display what changes can be made: sudo kubeadm upgrade plan Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': COMPONENT CURRENT AVAILABLE kubelet 1 x v1.19.0 v1.20.2 Upgrade to the latest stable version: COMPONENT CURRENT AVAILABLE kube-apiserver v1.19.7 v1.20.2 kube-controller-manager v1.19.7 v1.20.2 kube-scheduler v1.19.7 v1.20.2 kube-proxy v1.19.7 v1.20.2 CoreDNS 1.7.0 1.7.0 etcd 3.4.9-1 3.4.13-0 You can now apply the upgrade by executing the following command: kubeadm upgrade apply v1.20.2 Important note: kubelet must be upgraded manually after this step. Upgrade the cluster: kubeadm upgrade apply v1.20.2 upgrade Kubelet: sudo apt-get install --only-upgrade kubelet kubectl","title":"Perform a version upgrade on a Kubernetes cluster using Kubeadm"},{"location":"revision-topics/01-architcture-installation-configuration/#implement-etcd-backup-and-restore","text":"","title":"Implement etcd backup and restore"},{"location":"revision-topics/01-architcture-installation-configuration/#backing-up-etcd","text":"Take a snapshot of the DB, then store it in a safe location: ETCDCTL_API=3 etcdctl snapshot save snapshot.db --cacert /etc/kubernetes/pki/etcd/server.crt --cert /etc/kubernetes/pki/etcd/ca.crt --key /etc/kubernetes/pki/etcd/ca.key Verify the backup: sudo ETCDCTL_API=3 etcdctl --write-out=table snapshot status snapshot.db +----------+----------+------------+------------+ | HASH | REVISION | TOTAL KEYS | TOTAL SIZE | +----------+----------+------------+------------+ | 2125d542 | 364069 | 770 | 3.8 MB | +----------+----------+------------+------------+","title":"Backing up etcd"},{"location":"revision-topics/01-architcture-installation-configuration/#restore-to-etcd","text":"To perform a restore: ETCDCTL_API=3 etcdctl snapshot restore snapshot.db","title":"Restore to etcd"},{"location":"revision-topics/02-workloads-and-scheduling/","text":"Workloads and Scheduling Understand deployments and how to perform rolling update and rollbacks Deployments are intended to replace Replication Controllers. They provide the same replication functions (through Replica Sets) and also the ability to rollout changes and roll them back if necessary. An example configuration is shown below: apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 5 template: metadata: labels: app: nginx-frontend spec: containers: - name: nginx image: nginx:1.14 ports: - containerPort: 80 The main reason why leverage deployments is to manage a number of identical pods via one administrative unit - the deployment object. Should we need to make changes, we apply this to the deployment object, not individual pods. Because of the declarative nature of deployments , Kubernetes will rectify any changes between desired and running state, and rectify accordingly. For example, if we manually deleted. We can then describe it with kubectl describe deployment nginx-deployment Update To update an existing deployment, we have two main options: Rolling Update Recreate A rolling update, as the name implies, will swap out containers in a deployment with one created by a new image. Use a rolling update when the application supports having a mix of different pods (aka application versions). This method will also involve no downtime of the service, but will take longer to bring up the deployment to the requested version. Old and new versions of the pod spec will coexist until they're all rotated. A recreation will delete all the existing pods and then spin up new ones. This method will involve downtime. Consider this a \u201cbing bang\u201d approach Examples listed in the Kubernetes documentation are largely imperative, but I prefer to be declarative. As an example, create a new yaml file and make the required changes, in this example, the version of the nginx container is incremented. apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 5 template: metadata: labels: app: nginx-frontend spec: containers: - name: nginx image: nginx:1.15 ports: - containerPort: 80 We can then apply this file kubectl apply -f updateddeployment.yaml --record=true Followed by the following: kubectl rollout status deployment/nginx-deployment Waiting for deployment \"nginx-deployment\" rollout to finish: 2 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 2 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 2 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 2 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 3 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 3 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 3 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 3 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 4 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 4 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 4 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 4 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 4 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 1 old replicas are pending termination... Waiting for deployment \"nginx-deployment\" rollout to finish: 1 old replicas are pending termination... Waiting for deployment \"nginx-deployment\" rollout to finish: 1 old replicas are pending termination... Waiting for deployment \"nginx-deployment\" rollout to finish: 4 of 5 updated replicas are available... deployment \"nginx-deployment\" successfully rolled out We can also use the kubectl rollout history to look at the revision history of a deployment kubectl rollout history deployment/nginx-deployment deployment.extensions/nginx-deployment REVISION CHANGE-CAUSE 1 <none> 2 <none> 4 <none> 5 kubectl apply --filename=updateddeployment.yaml --record=true Alternatively, we can also do this imperatively: kubectl --record deployments/nginx-deployment set image deployments/nginx-deployment nginx=nginx:1.9.1 deployment.extensions/nginx-deployment image updated deployment.extensions/nginx-deployment image updated Rollback To rollback to the previous version: kubectl rollout undo deployment/nginx-deployment To rollback to a specific version: kubectl rollout undo deployment/nginx-deployment --to-revision 5 Source of revision : kubectl rollout history deployment/nginx-deployment Use ConfigMaps and Secrets to configure applications Configmaps are a way to decouple configuration from a pod manifest. Obviously, the first step is to create a config map before we can get pods to use them: kubectl create configmap <map-name> <data-source> \u201cMap-name\u201d is an arbitrary name we give to this particular map, and \u201cdata-source\u201d corresponds to a key-value pair that resides in the config map. kubectl create configmap vt-cm --from-literal=blog=virtualthoughts.co.uk At which point we can then describe it: kubectl describe configmap vt-cm Name: vt-cm Namespace: default Labels: <none> Annotations: <none> Data ==== blog: ---- virtualthoughts.co.uk To reference this config map in a pod, we declare it in the respective yaml: Configmaps can be mounted as volumes or environment variables . The below example leverages the latter. apiVersion: v1 kind: Pod metadata: name: config-test-pod spec: containers: - name: test-container image: busybox command: [ \"/bin/sh\", \"-c\", \"env\" ] env: - name: BLOG_NAME valueFrom: configMapKeyRef: name: vt-cm key: blog restartPolicy: Never The pod above will output the environment variables, so we can validate it\u2019s leveraged the config map by extracting the logs from the pod: kubectl logs config-test-pod | grep \"BLOG_NAME=\" ... BLOG_NAME=virtualthoughts.co.uk ... Know how to scale applications Constantly adding more, individual pods is not a sustainable model for scaling an application. To facilitate applications at scale, we need to leverage higher level constructs such as replicasets or deployments. As mentioned previously, deployments provide us with a single administrative unit to manage the underlying pods. We can scale a deployment object to increase the number of pods . As an example, if the following is deployed: apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 5 template: metadata: labels: app: nginx-frontend spec: containers: - name: nginx image: nginx:1.14 ports: - containerPort: 80 If we wanted to scale this, we can simply modify the yaml file and scale up/down the deployment by modifying the \u201creplicas\u201d field, or modify it in the fly: kubectl scale deployment nginx-deployment --replicas 10 Understand the primitives used to create robust, self-healing, application deployments Deployments facilitate this by employing a reconciliation loop to check the number of deployed pods matches what\u2019s defined in the manifest. Under the hood, deployments leverage ReplicaSets, which are primarily responsible for this feature. Stateful Sets are similar to deployments, for example they manage the deployment and scaling of a series of pods. However, in addition to deployments they also provide guarantees about the ordering and uniqueness of Pods. A StatefulSet maintains a sticky identity for each of their Pods. These pods are created from the same spec, but are not interchangeable: each has a persistent identifier that it maintains across any rescheduling. StatefulSets are valuable for applications that require one or more of the following. Stable, unique network identifiers. Stable, persistent storage. Ordered, graceful deployment and scaling. Ordered, automated rolling updates. apiVersion: apps/v1 kind: StatefulSet metadata: name: nginx-statefulset spec: selector: matchLabels: app: vt-nginx serviceName: \"nginx\" replicas: 2 template: metadata: labels: app: vt-nginx spec: containers: - name: vt-nginx image: nginx:1.7.9 ports: - containerPort: 80 Understand how resource limits can affect Pod scheduling At a namespace level, we can define resource limits. This enables a restriction in resources, especially helpful in multi-tenancy environments and provides a mechanism to prevent pods from consuming more resources than permitted, which may have a detrimental effect on the environment as a whole. We can define the following: Default memory / CPU requests & limits for a namespace Minimum and Maximum memory / CPU constraints for a namespace Memory/CPU Quotas for a namespace Default Requests and Limits If a container is created in a namespace with a default request/limit value and doesn't explicitly define these in the manifest, it inherits these values from the namespace Note, if you define a container with a memory/CPU limit, but not a request, Kubernetes will define the limit the same as the request. Minimum / Maximum Constraints If a pod does not meet the range in which the constraints are valued at, it will not be scheduled. Quotas Control the total amount of CPU/memory that can be consumed in the namespace as a whole. Example: Attempt to schedule a pod that request more memory than defined in the namespace Create a namespace: kubectl create namespace tenant-mem-limited Create a YAML manifest to limit resources: apiVersion: v1 kind: LimitRange metadata: name: tenant-max-mem namespace: tenant-mem-limited spec: limits: - max: memory: 250Mi type: Container Apply this to the aforementioned namespace: kubectl apply -f maxmem.yaml To create a pod with a memory request that exceeds the limit: apiVersion: v1 kind: Pod metadata: name: too-much-memory namespace: tenant-mem-limited spec: containers: - name: too-much-mem image: nginx resources: requests: memory: \"300Mi\" Executing the above will yield the following result: The Pod \"too-much-memory\" is invalid: spec.containers[0].resources.requests: Invalid value: \"300Mi\": must be less than or equal to memory limit As we have defined the pod limit of the namespace to 250MiB, a request for 300MiB will fail. Awareness of manifest management and common templating tools Kustomize Kustomize is a templating tool for Kubernetes manifests in its native form (Yaml). When working with raw YAML files you will typically have a directory containing several files identifying the resources it creates. To begin, a directory containing our manifests needs to exist: /home/david/app/base total 16 drwxrwxr-x 2 david david 4096 Feb 9 11:44 . drwxr-xr-x 27 david david 4096 Feb 9 11:44 .. -rw-rw-r-- 1 david david 340 Feb 9 11:09 deployment.yaml -rw-rw-r-- 1 david david 153 Feb 9 11:09 service.yaml This will form our base - we will build on this but adding customisations in the form of overlays. First, we need a kustomize file. which can be created with kustomize create --autodetect This will create kustomization.yaml in the current directory: total 20 drwxrwxr-x 2 david david 4096 Feb 9 11:47 . drwxr-xr-x 27 david david 4096 Feb 9 11:47 .. -rw-rw-r-- 1 david david 340 Feb 9 11:09 deployment.yaml -rw-rw-r-- 1 david david 108 Feb 9 11:47 kustomization.yaml -rw-rw-r-- 1 david david 153 Feb 9 11:09 service.yaml The contents being: apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - deployment.yaml - service.yaml Variants and Overlays variant - Divergence in configuration from the base overlay - Composes variants together Say, for example, we wanted to generate manifests for different environments (prod and dev) that are based from this config, but have additional customisations. In this example we will create a dev variant encapsulated in a single Overlay mkdir -p overlays/{dev,prod} cd overlays/dev Begin by creating a Kustomization object specifying the base (this will create kustomization.yaml ) : kustomize create --resources ../../base In this example, I want to change the replica count to 1, as it's a dev environment. In the dev directory, create a new file deployment.yaml containing: apiVersion: apps/v1 kind: Deployment metadata: labels: app: nginx name: nginx-deployment spec: replicas: 1 The kustomization.yaml file needs modifying to include a patchesStrategicMerge block: apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization bases: - ../../base patchesStrategicMerge: - deployment.yaml Patches can be used to apply different customizations to Resources. Kustomize supports different patching mechanisms through patchesStrategicMerge and patchesJson6902 . patchesStrategicMerge is a list of file paths. We can generate the manifests and apply to the cluster by executing (from the base folder): kustomize build ./overlay/dev | kubectl apply -f - By running this, only 1 pod will be created in the deployment object, instead of what's defined in the base because of the customisation we've applied. We can do the same with prod, or any arbitrary number of environments. Helm Helm is synonymous to what apt or yum are in the Linux world. It's effectively a package manager for Kubernetes. \"Packages\" in Helm are called charts to which you can customise with your own values. It's unlikely the exam will require anyone to create a helm chart from scratch, but an understanding of how it works is a good idea. Helm Repos Repos are where helm charts are stored. Typically, a repo will contain a number of charts to choose from. Helm can be managed by a CLI client, and a repo can be added by running: helm repo add bitnami https://charts.bitnami.com/bitnami To list the packages from this repo: helm search repo bitnami To install a package from this repo: helm install my-release bitnami/mariadb Where my-release is a string identifying an installed instance of this application Parameters that can be customised - are dependent on how the chart is configured. For the aforementioned MariaDB chart, they are listed at https://github.com/bitnami/charts/tree/master/bitnami/mariadb/#parameters These values are encapsulated in the corresponding values.yaml file in the repo. You can populate an instance of it and apply it with: helm install -f https://raw.githubusercontent.com/bitnami/charts/master/bitnami/mariadb/values.yaml my-release bitnami/mariadb Alternatively, variables can be declared by using --set , such as: helm install my-release --set auth.rootPassword=secretpassword bitnami/mariadb","title":"Part Two - Workloads & Scheduling"},{"location":"revision-topics/02-workloads-and-scheduling/#workloads-and-scheduling","text":"","title":"Workloads and Scheduling"},{"location":"revision-topics/02-workloads-and-scheduling/#understand-deployments-and-how-to-perform-rolling-update-and-rollbacks","text":"Deployments are intended to replace Replication Controllers. They provide the same replication functions (through Replica Sets) and also the ability to rollout changes and roll them back if necessary. An example configuration is shown below: apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 5 template: metadata: labels: app: nginx-frontend spec: containers: - name: nginx image: nginx:1.14 ports: - containerPort: 80 The main reason why leverage deployments is to manage a number of identical pods via one administrative unit - the deployment object. Should we need to make changes, we apply this to the deployment object, not individual pods. Because of the declarative nature of deployments , Kubernetes will rectify any changes between desired and running state, and rectify accordingly. For example, if we manually deleted. We can then describe it with kubectl describe deployment nginx-deployment","title":"Understand deployments and how to perform rolling update and rollbacks"},{"location":"revision-topics/02-workloads-and-scheduling/#update","text":"To update an existing deployment, we have two main options: Rolling Update Recreate A rolling update, as the name implies, will swap out containers in a deployment with one created by a new image. Use a rolling update when the application supports having a mix of different pods (aka application versions). This method will also involve no downtime of the service, but will take longer to bring up the deployment to the requested version. Old and new versions of the pod spec will coexist until they're all rotated. A recreation will delete all the existing pods and then spin up new ones. This method will involve downtime. Consider this a \u201cbing bang\u201d approach Examples listed in the Kubernetes documentation are largely imperative, but I prefer to be declarative. As an example, create a new yaml file and make the required changes, in this example, the version of the nginx container is incremented. apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 5 template: metadata: labels: app: nginx-frontend spec: containers: - name: nginx image: nginx:1.15 ports: - containerPort: 80 We can then apply this file kubectl apply -f updateddeployment.yaml --record=true Followed by the following: kubectl rollout status deployment/nginx-deployment Waiting for deployment \"nginx-deployment\" rollout to finish: 2 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 2 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 2 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 2 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 3 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 3 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 3 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 3 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 4 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 4 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 4 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 4 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 4 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 1 old replicas are pending termination... Waiting for deployment \"nginx-deployment\" rollout to finish: 1 old replicas are pending termination... Waiting for deployment \"nginx-deployment\" rollout to finish: 1 old replicas are pending termination... Waiting for deployment \"nginx-deployment\" rollout to finish: 4 of 5 updated replicas are available... deployment \"nginx-deployment\" successfully rolled out We can also use the kubectl rollout history to look at the revision history of a deployment kubectl rollout history deployment/nginx-deployment deployment.extensions/nginx-deployment REVISION CHANGE-CAUSE 1 <none> 2 <none> 4 <none> 5 kubectl apply --filename=updateddeployment.yaml --record=true Alternatively, we can also do this imperatively: kubectl --record deployments/nginx-deployment set image deployments/nginx-deployment nginx=nginx:1.9.1 deployment.extensions/nginx-deployment image updated deployment.extensions/nginx-deployment image updated","title":"Update"},{"location":"revision-topics/02-workloads-and-scheduling/#rollback","text":"To rollback to the previous version: kubectl rollout undo deployment/nginx-deployment To rollback to a specific version: kubectl rollout undo deployment/nginx-deployment --to-revision 5 Source of revision : kubectl rollout history deployment/nginx-deployment","title":"Rollback"},{"location":"revision-topics/02-workloads-and-scheduling/#use-configmaps-and-secrets-to-configure-applications","text":"Configmaps are a way to decouple configuration from a pod manifest. Obviously, the first step is to create a config map before we can get pods to use them: kubectl create configmap <map-name> <data-source> \u201cMap-name\u201d is an arbitrary name we give to this particular map, and \u201cdata-source\u201d corresponds to a key-value pair that resides in the config map. kubectl create configmap vt-cm --from-literal=blog=virtualthoughts.co.uk At which point we can then describe it: kubectl describe configmap vt-cm Name: vt-cm Namespace: default Labels: <none> Annotations: <none> Data ==== blog: ---- virtualthoughts.co.uk To reference this config map in a pod, we declare it in the respective yaml: Configmaps can be mounted as volumes or environment variables . The below example leverages the latter. apiVersion: v1 kind: Pod metadata: name: config-test-pod spec: containers: - name: test-container image: busybox command: [ \"/bin/sh\", \"-c\", \"env\" ] env: - name: BLOG_NAME valueFrom: configMapKeyRef: name: vt-cm key: blog restartPolicy: Never The pod above will output the environment variables, so we can validate it\u2019s leveraged the config map by extracting the logs from the pod: kubectl logs config-test-pod | grep \"BLOG_NAME=\" ... BLOG_NAME=virtualthoughts.co.uk ...","title":"Use ConfigMaps and Secrets to configure applications"},{"location":"revision-topics/02-workloads-and-scheduling/#know-how-to-scale-applications","text":"Constantly adding more, individual pods is not a sustainable model for scaling an application. To facilitate applications at scale, we need to leverage higher level constructs such as replicasets or deployments. As mentioned previously, deployments provide us with a single administrative unit to manage the underlying pods. We can scale a deployment object to increase the number of pods . As an example, if the following is deployed: apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 5 template: metadata: labels: app: nginx-frontend spec: containers: - name: nginx image: nginx:1.14 ports: - containerPort: 80 If we wanted to scale this, we can simply modify the yaml file and scale up/down the deployment by modifying the \u201creplicas\u201d field, or modify it in the fly: kubectl scale deployment nginx-deployment --replicas 10","title":"Know how to scale applications"},{"location":"revision-topics/02-workloads-and-scheduling/#understand-the-primitives-used-to-create-robust-self-healing-application-deployments","text":"Deployments facilitate this by employing a reconciliation loop to check the number of deployed pods matches what\u2019s defined in the manifest. Under the hood, deployments leverage ReplicaSets, which are primarily responsible for this feature. Stateful Sets are similar to deployments, for example they manage the deployment and scaling of a series of pods. However, in addition to deployments they also provide guarantees about the ordering and uniqueness of Pods. A StatefulSet maintains a sticky identity for each of their Pods. These pods are created from the same spec, but are not interchangeable: each has a persistent identifier that it maintains across any rescheduling. StatefulSets are valuable for applications that require one or more of the following. Stable, unique network identifiers. Stable, persistent storage. Ordered, graceful deployment and scaling. Ordered, automated rolling updates. apiVersion: apps/v1 kind: StatefulSet metadata: name: nginx-statefulset spec: selector: matchLabels: app: vt-nginx serviceName: \"nginx\" replicas: 2 template: metadata: labels: app: vt-nginx spec: containers: - name: vt-nginx image: nginx:1.7.9 ports: - containerPort: 80","title":"Understand the primitives used to create robust, self-healing, application deployments"},{"location":"revision-topics/02-workloads-and-scheduling/#understand-how-resource-limits-can-affect-pod-scheduling","text":"At a namespace level, we can define resource limits. This enables a restriction in resources, especially helpful in multi-tenancy environments and provides a mechanism to prevent pods from consuming more resources than permitted, which may have a detrimental effect on the environment as a whole. We can define the following: Default memory / CPU requests & limits for a namespace Minimum and Maximum memory / CPU constraints for a namespace Memory/CPU Quotas for a namespace","title":"Understand how resource limits can affect Pod scheduling"},{"location":"revision-topics/02-workloads-and-scheduling/#default-requests-and-limits","text":"If a container is created in a namespace with a default request/limit value and doesn't explicitly define these in the manifest, it inherits these values from the namespace Note, if you define a container with a memory/CPU limit, but not a request, Kubernetes will define the limit the same as the request.","title":"Default Requests and Limits"},{"location":"revision-topics/02-workloads-and-scheduling/#minimum-maximum-constraints","text":"If a pod does not meet the range in which the constraints are valued at, it will not be scheduled.","title":"Minimum / Maximum Constraints"},{"location":"revision-topics/02-workloads-and-scheduling/#quotas","text":"Control the total amount of CPU/memory that can be consumed in the namespace as a whole. Example: Attempt to schedule a pod that request more memory than defined in the namespace Create a namespace: kubectl create namespace tenant-mem-limited Create a YAML manifest to limit resources: apiVersion: v1 kind: LimitRange metadata: name: tenant-max-mem namespace: tenant-mem-limited spec: limits: - max: memory: 250Mi type: Container Apply this to the aforementioned namespace: kubectl apply -f maxmem.yaml To create a pod with a memory request that exceeds the limit: apiVersion: v1 kind: Pod metadata: name: too-much-memory namespace: tenant-mem-limited spec: containers: - name: too-much-mem image: nginx resources: requests: memory: \"300Mi\" Executing the above will yield the following result: The Pod \"too-much-memory\" is invalid: spec.containers[0].resources.requests: Invalid value: \"300Mi\": must be less than or equal to memory limit As we have defined the pod limit of the namespace to 250MiB, a request for 300MiB will fail.","title":"Quotas"},{"location":"revision-topics/02-workloads-and-scheduling/#awareness-of-manifest-management-and-common-templating-tools","text":"","title":"Awareness of manifest management and common templating tools"},{"location":"revision-topics/02-workloads-and-scheduling/#kustomize","text":"Kustomize is a templating tool for Kubernetes manifests in its native form (Yaml). When working with raw YAML files you will typically have a directory containing several files identifying the resources it creates. To begin, a directory containing our manifests needs to exist: /home/david/app/base total 16 drwxrwxr-x 2 david david 4096 Feb 9 11:44 . drwxr-xr-x 27 david david 4096 Feb 9 11:44 .. -rw-rw-r-- 1 david david 340 Feb 9 11:09 deployment.yaml -rw-rw-r-- 1 david david 153 Feb 9 11:09 service.yaml This will form our base - we will build on this but adding customisations in the form of overlays. First, we need a kustomize file. which can be created with kustomize create --autodetect This will create kustomization.yaml in the current directory: total 20 drwxrwxr-x 2 david david 4096 Feb 9 11:47 . drwxr-xr-x 27 david david 4096 Feb 9 11:47 .. -rw-rw-r-- 1 david david 340 Feb 9 11:09 deployment.yaml -rw-rw-r-- 1 david david 108 Feb 9 11:47 kustomization.yaml -rw-rw-r-- 1 david david 153 Feb 9 11:09 service.yaml The contents being: apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - deployment.yaml - service.yaml","title":"Kustomize"},{"location":"revision-topics/02-workloads-and-scheduling/#variants-and-overlays","text":"variant - Divergence in configuration from the base overlay - Composes variants together Say, for example, we wanted to generate manifests for different environments (prod and dev) that are based from this config, but have additional customisations. In this example we will create a dev variant encapsulated in a single Overlay mkdir -p overlays/{dev,prod} cd overlays/dev Begin by creating a Kustomization object specifying the base (this will create kustomization.yaml ) : kustomize create --resources ../../base In this example, I want to change the replica count to 1, as it's a dev environment. In the dev directory, create a new file deployment.yaml containing: apiVersion: apps/v1 kind: Deployment metadata: labels: app: nginx name: nginx-deployment spec: replicas: 1 The kustomization.yaml file needs modifying to include a patchesStrategicMerge block: apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization bases: - ../../base patchesStrategicMerge: - deployment.yaml Patches can be used to apply different customizations to Resources. Kustomize supports different patching mechanisms through patchesStrategicMerge and patchesJson6902 . patchesStrategicMerge is a list of file paths. We can generate the manifests and apply to the cluster by executing (from the base folder): kustomize build ./overlay/dev | kubectl apply -f - By running this, only 1 pod will be created in the deployment object, instead of what's defined in the base because of the customisation we've applied. We can do the same with prod, or any arbitrary number of environments.","title":"Variants and Overlays"},{"location":"revision-topics/02-workloads-and-scheduling/#helm","text":"Helm is synonymous to what apt or yum are in the Linux world. It's effectively a package manager for Kubernetes. \"Packages\" in Helm are called charts to which you can customise with your own values. It's unlikely the exam will require anyone to create a helm chart from scratch, but an understanding of how it works is a good idea.","title":"Helm"},{"location":"revision-topics/02-workloads-and-scheduling/#helm-repos","text":"Repos are where helm charts are stored. Typically, a repo will contain a number of charts to choose from. Helm can be managed by a CLI client, and a repo can be added by running: helm repo add bitnami https://charts.bitnami.com/bitnami To list the packages from this repo: helm search repo bitnami To install a package from this repo: helm install my-release bitnami/mariadb Where my-release is a string identifying an installed instance of this application Parameters that can be customised - are dependent on how the chart is configured. For the aforementioned MariaDB chart, they are listed at https://github.com/bitnami/charts/tree/master/bitnami/mariadb/#parameters These values are encapsulated in the corresponding values.yaml file in the repo. You can populate an instance of it and apply it with: helm install -f https://raw.githubusercontent.com/bitnami/charts/master/bitnami/mariadb/values.yaml my-release bitnami/mariadb Alternatively, variables can be declared by using --set , such as: helm install my-release --set auth.rootPassword=secretpassword bitnami/mariadb","title":"Helm Repos"},{"location":"revision-topics/03-services-and-networking/","text":"Services & Networking Understand host networking configuration on the cluster nodes At the host level, we have an interface (typically something like eth0 or ens192 etc) that acts as the primary network adapter. Each host is responsible for one subnet of the CNI range. In this example, the left host is responsible for 10.1.1.0/24, and the right host 10.1.2.0/24. The overall pod CIDR block may be something like 10.1.0.0/16. Virtual ethernet adapters are paired with a corresponding Pod network adapter. Kernel routing is used to enable Pods to communicate outside the host it resides in. Understand connectivity between Pods Every Pod gets its own IP address. This means you do not need to explicitly create links between Pods, and you almost never need to deal with mapping container ports to host ports. This creates a clean, backwards-compatible model where Pods can be treated much like VMs or physical hosts from the perspectives of port allocation, naming, service discovery, load balancing, application configuration, and migration. Kubernetes imposes the following fundamental requirements on any networking implementation (barring any intentional network segmentation policies): Pods on a node can communicate with all pods on all nodes without NAT Agents on a node (e.g. system daemons, Kubelet) can communicate with all pods on that node Note: When running workloads that leverage hostNetwork : Pods in the host network of a node can communicate with all pods on all nodes without NAT Understand ClusterIP, NodePort, LoadBalancer service types and endpoints Pods are ephemeral. Therefore, placing these behind a service which provides a stable, static entrypoint is a fundamental use of the kubernetes service object. To reiterate, services take the form of the following: ClusterIP - Internal only LoadBalancer - External, requires cloud provider, or software implementation to provide one NodePort - External, requires access the nodes directly Ingress resource - L7 An Ingress can be configured to give services externally-reachable URLs, load balance traffic, terminate SSL, and offer name based virtual hosting. An Ingress controller is responsible for fulfilling the Ingress, usually with a loadbalancer, though it may also configure your edge router or additional frontends to help handle the traffic. Know how to use Ingress controllers and Ingress resources Ingress exposes HTTP and HTTPS routes from outside the cluster to services within a cluster. Ingress consists of two components. Ingress Resource is a collection of rules for the inbound traffic to reach Services. These are Layer 7 (L7) rules that allow hostnames (and optionally paths) to be directed to specific Services in Kubernetes. The second component is the Ingress Controller which acts upon the rules set by the Ingress Resource, typically via an HTTP or L7 load balancer. It is vital that both pieces are properly configured to route traffic from an outside client to a Kubernetes Service. The following yaml creates two ingress rules for the website foo.bar.com The default path will direct traffic to the service \u201cdefault-service\u201d which listens on port 80 Paths ending in /foo will direct traffic to the service \u201cservice1\u201d which listens on port 4200 Paths ending in /bar will direct traffic to the service \u201cservice2\u201d which listens on port 8080 apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: simple-fanout-example spec: rules: - host: foo.bar.com http: paths: - path: / pathType: Prefix backend: service: name: default-service port: number: 80 - path: /foo pathType: Prefix backend: service: name: service1 port: number: 4200 - path: /bar pathType: Prefix backend: service: name: service2 port: number: 8080 In order for the Ingress resource to work, the cluster must have an ingress controller running. Ingress controllers are deployed into the Kubernetes cluster as a workload: > kubectl get po -A | grep nginx-ingress ingress-nginx nginx-ingress-controller-2gxtd 1/1 Running 0 14d ingress-nginx nginx-ingress-controller-9lrzh 1/1 Running 0 14d ingress-nginx nginx-ingress-controller-r2ksq 1/1 Running 0 14d Know how to configure and use CoreDNS As of 1.13, coredns has replaced kube-dns as the facilitator of cluster DNS and runs as pods. kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-fb8b8dccf-hxbhn 1/1 Running 9 10d coredns-fb8b8dccf-jks6g 1/1 Running 4 8d To view the DNS configuration of a pod, spin one up and inspect the /etc/resolv.conf: kubectl run busybox --image=busybox -- sleep 9000 kubectl exec -it busybox sh / # cat /etc/resolv.conf nameserver 10.96.0.10 search default.svc.cluster.local svc.cluster.local cluster.local virtualthoughts.co.uk options ndots:5 \u201c10.96.0.10\u201d references the Kube-DNS service \u201cDefault.svc.cluster.local\u201d References the namespace with the suffix svc.cluster.local. All pods are provisioned a DNS record and are in the format of [Pod IP separated by dashes].[Namespace].[type].[Base Domain Name] Where [type] is pod in this example, put services can be resolved by the same convention. For example: / # nslookup 10-42-2-68.default.pod.cluster.local Server: 10.43.0.10 Address: 10.43.0.10:53 Name: 10-42-2-68.default.pod.cluster.local Address: 10.42.2.68 Services follow a similar pattern [Service Name].[Namespace].[type].[Base Domain Name] For example: my-svc.my-namespace.svc.cluster-domain.example Headless services are those without a cluster ip, but will respond with a list of IP\u2019s of pods that are applicable at that particular moment in time. apiVersion: v1 kind: Service metadata: name: test-headless spec: clusterIP: None ports: - port: 80 targetPort: 80 selector: app: web-headless We can modify the default behavior of the pod dns configuration in the yaml file: apiVersion: v1 kind: Pod metadata: namespace: default name: dns-example spec: containers: - name: test image: nginx dnsPolicy: \"None\" dnsConfig: nameservers: - 8.8.8.8 searches: - ns1.svc.cluster.local - my.dns.search.suffix options: - name: ndots value: \"2\" - name: edns0 CoreDNS also has a configmap that can be modified: kubectl get cm coredns -n kube-system -o yaml apiVersion: v1 data: Corefile: | .:53 { errors health ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure upstream fallthrough in-addr.arpa ip6.arpa } hosts /etc/coredns/NodeHosts { reload 1s fallthrough } prometheus :9153 forward . /etc/resolv.conf cache 30 loop reload loadbalance } NodeHosts: | 172.16.10.100 k3s-ranch-node-1 172.16.10.101 k3s-ranch-node-2 172.16.10.102 k3s-ranch-node-3 The Corefile configuration includes the following plugins of CoreDNS: errors : Errors are logged to stdout. health : Health of CoreDNS is reported to http://localhost:8080/health . In this extended syntax lameduck will make the process unhealthy then wait for 5 seconds before the process is shut down. ready : An HTTP endpoint on port 8181 will return 200 OK, when all plugins that are able to signal readiness have done so. kubernetes : CoreDNS will reply to DNS queries based on IP of the services and pods of Kubernetes. You can find more details about that plugin on the CoreDNS website. ttl allows you to set a custom TTL for responses. The default is 5 seconds. The minimum TTL allowed is 0 seconds, and the maximum is capped at 3600 seconds. Setting TTL to 0 will prevent records from being cached. The pods insecure option is provided for backward compatibility with kube-dns. You can use the pods verified option, which returns an A record only if there exists a pod in same namespace with matching IP. The pods disabled option can be used if you don't use pod records. prometheus : Metrics of CoreDNS are available at http://localhost:9153/metrics in Prometheus format (also known as OpenMetrics). forward : Any queries that are not within the cluster domain of Kubernetes will be forwarded to predefined resolvers (/etc/resolv.conf). cache: This enables a frontend cache. loop : Detects simple forwarding loops and halts the CoreDNS process if a loop is found. reload : Allows automatic reload of a changed Corefile. After you edit the ConfigMap configuration, allow two minutes for your changes to take effect. You can modify the default CoreDNS behavior by modifying the ConfigMap. Choose an appropriate container network interface plugin You must deploy a Container Network Interface (CNI) based Pod network add-on so that your Pods can communicate with each other. Cluster DNS (CoreDNS) will not start up before a network is installed. https://kubernetes.io/docs/concepts/cluster-administration/addons/#networking-and-network-policy As a generalisation, CNI\u2019s provide some kind of network overlay. But each have their own features, limitations and considerations. CNI's manifest and Kubernetes pods in your cluster. A typical workflow would involve standing up your k8s cluster and applying a network cni with: kubectl apply -f <add-on.yaml>","title":"Part Three - Services & Networking"},{"location":"revision-topics/03-services-and-networking/#services-networking","text":"","title":"Services &amp; Networking"},{"location":"revision-topics/03-services-and-networking/#understand-host-networking-configuration-on-the-cluster-nodes","text":"At the host level, we have an interface (typically something like eth0 or ens192 etc) that acts as the primary network adapter. Each host is responsible for one subnet of the CNI range. In this example, the left host is responsible for 10.1.1.0/24, and the right host 10.1.2.0/24. The overall pod CIDR block may be something like 10.1.0.0/16. Virtual ethernet adapters are paired with a corresponding Pod network adapter. Kernel routing is used to enable Pods to communicate outside the host it resides in.","title":"Understand host networking configuration on the cluster nodes"},{"location":"revision-topics/03-services-and-networking/#understand-connectivity-between-pods","text":"Every Pod gets its own IP address. This means you do not need to explicitly create links between Pods, and you almost never need to deal with mapping container ports to host ports. This creates a clean, backwards-compatible model where Pods can be treated much like VMs or physical hosts from the perspectives of port allocation, naming, service discovery, load balancing, application configuration, and migration. Kubernetes imposes the following fundamental requirements on any networking implementation (barring any intentional network segmentation policies): Pods on a node can communicate with all pods on all nodes without NAT Agents on a node (e.g. system daemons, Kubelet) can communicate with all pods on that node Note: When running workloads that leverage hostNetwork : Pods in the host network of a node can communicate with all pods on all nodes without NAT","title":"Understand connectivity between Pods"},{"location":"revision-topics/03-services-and-networking/#understand-clusterip-nodeport-loadbalancer-service-types-and-endpoints","text":"Pods are ephemeral. Therefore, placing these behind a service which provides a stable, static entrypoint is a fundamental use of the kubernetes service object. To reiterate, services take the form of the following: ClusterIP - Internal only LoadBalancer - External, requires cloud provider, or software implementation to provide one NodePort - External, requires access the nodes directly Ingress resource - L7 An Ingress can be configured to give services externally-reachable URLs, load balance traffic, terminate SSL, and offer name based virtual hosting. An Ingress controller is responsible for fulfilling the Ingress, usually with a loadbalancer, though it may also configure your edge router or additional frontends to help handle the traffic.","title":"Understand ClusterIP, NodePort, LoadBalancer service types and endpoints"},{"location":"revision-topics/03-services-and-networking/#know-how-to-use-ingress-controllers-and-ingress-resources","text":"Ingress exposes HTTP and HTTPS routes from outside the cluster to services within a cluster. Ingress consists of two components. Ingress Resource is a collection of rules for the inbound traffic to reach Services. These are Layer 7 (L7) rules that allow hostnames (and optionally paths) to be directed to specific Services in Kubernetes. The second component is the Ingress Controller which acts upon the rules set by the Ingress Resource, typically via an HTTP or L7 load balancer. It is vital that both pieces are properly configured to route traffic from an outside client to a Kubernetes Service. The following yaml creates two ingress rules for the website foo.bar.com The default path will direct traffic to the service \u201cdefault-service\u201d which listens on port 80 Paths ending in /foo will direct traffic to the service \u201cservice1\u201d which listens on port 4200 Paths ending in /bar will direct traffic to the service \u201cservice2\u201d which listens on port 8080 apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: simple-fanout-example spec: rules: - host: foo.bar.com http: paths: - path: / pathType: Prefix backend: service: name: default-service port: number: 80 - path: /foo pathType: Prefix backend: service: name: service1 port: number: 4200 - path: /bar pathType: Prefix backend: service: name: service2 port: number: 8080 In order for the Ingress resource to work, the cluster must have an ingress controller running. Ingress controllers are deployed into the Kubernetes cluster as a workload: > kubectl get po -A | grep nginx-ingress ingress-nginx nginx-ingress-controller-2gxtd 1/1 Running 0 14d ingress-nginx nginx-ingress-controller-9lrzh 1/1 Running 0 14d ingress-nginx nginx-ingress-controller-r2ksq 1/1 Running 0 14d","title":"Know how to use Ingress controllers and Ingress resources"},{"location":"revision-topics/03-services-and-networking/#know-how-to-configure-and-use-coredns","text":"As of 1.13, coredns has replaced kube-dns as the facilitator of cluster DNS and runs as pods. kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-fb8b8dccf-hxbhn 1/1 Running 9 10d coredns-fb8b8dccf-jks6g 1/1 Running 4 8d To view the DNS configuration of a pod, spin one up and inspect the /etc/resolv.conf: kubectl run busybox --image=busybox -- sleep 9000 kubectl exec -it busybox sh / # cat /etc/resolv.conf nameserver 10.96.0.10 search default.svc.cluster.local svc.cluster.local cluster.local virtualthoughts.co.uk options ndots:5 \u201c10.96.0.10\u201d references the Kube-DNS service \u201cDefault.svc.cluster.local\u201d References the namespace with the suffix svc.cluster.local. All pods are provisioned a DNS record and are in the format of [Pod IP separated by dashes].[Namespace].[type].[Base Domain Name] Where [type] is pod in this example, put services can be resolved by the same convention. For example: / # nslookup 10-42-2-68.default.pod.cluster.local Server: 10.43.0.10 Address: 10.43.0.10:53 Name: 10-42-2-68.default.pod.cluster.local Address: 10.42.2.68 Services follow a similar pattern [Service Name].[Namespace].[type].[Base Domain Name] For example: my-svc.my-namespace.svc.cluster-domain.example Headless services are those without a cluster ip, but will respond with a list of IP\u2019s of pods that are applicable at that particular moment in time. apiVersion: v1 kind: Service metadata: name: test-headless spec: clusterIP: None ports: - port: 80 targetPort: 80 selector: app: web-headless We can modify the default behavior of the pod dns configuration in the yaml file: apiVersion: v1 kind: Pod metadata: namespace: default name: dns-example spec: containers: - name: test image: nginx dnsPolicy: \"None\" dnsConfig: nameservers: - 8.8.8.8 searches: - ns1.svc.cluster.local - my.dns.search.suffix options: - name: ndots value: \"2\" - name: edns0 CoreDNS also has a configmap that can be modified: kubectl get cm coredns -n kube-system -o yaml apiVersion: v1 data: Corefile: | .:53 { errors health ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure upstream fallthrough in-addr.arpa ip6.arpa } hosts /etc/coredns/NodeHosts { reload 1s fallthrough } prometheus :9153 forward . /etc/resolv.conf cache 30 loop reload loadbalance } NodeHosts: | 172.16.10.100 k3s-ranch-node-1 172.16.10.101 k3s-ranch-node-2 172.16.10.102 k3s-ranch-node-3 The Corefile configuration includes the following plugins of CoreDNS: errors : Errors are logged to stdout. health : Health of CoreDNS is reported to http://localhost:8080/health . In this extended syntax lameduck will make the process unhealthy then wait for 5 seconds before the process is shut down. ready : An HTTP endpoint on port 8181 will return 200 OK, when all plugins that are able to signal readiness have done so. kubernetes : CoreDNS will reply to DNS queries based on IP of the services and pods of Kubernetes. You can find more details about that plugin on the CoreDNS website. ttl allows you to set a custom TTL for responses. The default is 5 seconds. The minimum TTL allowed is 0 seconds, and the maximum is capped at 3600 seconds. Setting TTL to 0 will prevent records from being cached. The pods insecure option is provided for backward compatibility with kube-dns. You can use the pods verified option, which returns an A record only if there exists a pod in same namespace with matching IP. The pods disabled option can be used if you don't use pod records. prometheus : Metrics of CoreDNS are available at http://localhost:9153/metrics in Prometheus format (also known as OpenMetrics). forward : Any queries that are not within the cluster domain of Kubernetes will be forwarded to predefined resolvers (/etc/resolv.conf). cache: This enables a frontend cache. loop : Detects simple forwarding loops and halts the CoreDNS process if a loop is found. reload : Allows automatic reload of a changed Corefile. After you edit the ConfigMap configuration, allow two minutes for your changes to take effect. You can modify the default CoreDNS behavior by modifying the ConfigMap.","title":"Know how to configure and use CoreDNS"},{"location":"revision-topics/03-services-and-networking/#choose-an-appropriate-container-network-interface-plugin","text":"You must deploy a Container Network Interface (CNI) based Pod network add-on so that your Pods can communicate with each other. Cluster DNS (CoreDNS) will not start up before a network is installed. https://kubernetes.io/docs/concepts/cluster-administration/addons/#networking-and-network-policy As a generalisation, CNI\u2019s provide some kind of network overlay. But each have their own features, limitations and considerations. CNI's manifest and Kubernetes pods in your cluster. A typical workflow would involve standing up your k8s cluster and applying a network cni with: kubectl apply -f <add-on.yaml>","title":"Choose an appropriate container network interface plugin"},{"location":"revision-topics/04-storage/","text":"Storage Understand storage classes, persistent volumes A StorageClass provides a way for administrators to describe the \"classes\" of storage they offer. Different classes might map to quality-of-service levels, or to backup policies, or to arbitrary policies determined by the cluster administrators. Kubernetes itself is un-opinionated about what classes represent. This concept is sometimes called \"profiles\" in other storage systems. An example of a storage class is below: apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: standard provisioner: kubernetes.io/aws-ebs parameters: type: gp2 reclaimPolicy: Retain allowVolumeExpansion: true mountOptions: - debug volumeBindingMode: Immediate Key declarations are: provisioner : Determine which volume plugin to use. This usually matches to a cloud provider, and a specific storage service that it offers. In this example AWS Elastic Block Store parameters : Describe characteristics of this storage class in context of the underlying provisioner. In this example, the type is gp2 which, in AWS lingo means General Purpose SSD. Other types include IO1 (Provisioned IOPS), ST1 (Throughput Optimised) and STC (Cold Storage). A storageclass object by itself defines what storage is provisioned when it is invoked . By itself, it does nothing. A persistentvolume object can be used to request storage from a storageclass and is typically part of a pod manifest apiVersion: v1 kind: PersistentVolume metadata: name: pv0003 spec: capacity: storage: 5Gi volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Recycle storageClassName: standard Understand volume mode, access modes and reclaim policies for volumes Volume modes Only two exist: block - Mounted to a pod as a raw block device without a filesystem. The Pod / application needs to understand how to deal with raw block devices. Presenting it in this way can yield better performance, at the expense of complexity. filesystem - Mounted inside a pods' filesystem inside a directory. If the volume is backed by a block device with no filesystem, Kubernetes will create one. Compared to block devices, this method offers the highest compatibility, at the expense of performance. Access Modes Three options exist: ReadWriteOnce \u2013 The volume can be mounted as read-write by a single node ReadOnlyMany \u2013 The volume can be mounted read-only by many nodes ReadWriteMany \u2013 The volume can be mounted as read-write by many nodes Understand persistent volume claims primitive A PersistentVolume can be thought of as storage provisioned by an administrator. Think of this as pre-allocation A PersistentVolumeClaim can be thought of as storage requested by a user/workload. When the user creates PVC to request storage, Kubernetes will try to match that PVC with a pre-allocated PV. If a match can be found, the PVC will be bound to the PV, and the user will start to use that pre-allocated piece of storage. Know how to configure applications with persistent storage When not using storageclasses there are a number of steps: Create the PV: apiVersion: v1 kind: PersistentVolume metadata: name: task-pv-volume labels: type: local spec: storageClassName: manual capacity: storage: 10Gi accessModes: - ReadWriteOnce hostPath: path: \"/mnt/data\" Create the workload to leverage this: apiVersion: v1 kind: PersistentVolumeClaim metadata: name: task-pv-claim spec: storageClassName: manual accessModes: - ReadWriteOnce resources: requests: storage: 3Gi --- apiVersion: v1 kind: Pod metadata: name: task-pv-pod spec: volumes: - name: task-pv-storage persistentVolumeClaim: claimName: task-pv-claim containers: - name: task-pv-container image: nginx ports: - containerPort: 80 name: \"http-server\" volumeMounts: - mountPath: \"/usr/share/nginx/html\" name: task-pv-storage If leveraging a storageclass, a persistentvolume object is not required, we just need a persistentvolumeclaim apiVersion: v1 kind: PersistentVolumeClaim metadata: name: task-pv-claim spec: storageClassName: myStorageClass accessModes: - ReadWriteOnce resources: requests: storage: 3Gi --- apiVersion: v1 kind: Pod metadata: name: task-pv-pod spec: volumes: - name: task-pv-storage persistentVolumeClaim: claimName: task-pv-claim containers: - name: task-pv-container image: nginx ports: - containerPort: 80 name: \"http-server\" volumeMounts: - mountPath: \"/usr/share/nginx/html\" name: task-pv-storage","title":"Part Four - Storage"},{"location":"revision-topics/04-storage/#storage","text":"","title":"Storage"},{"location":"revision-topics/04-storage/#understand-storage-classes-persistent-volumes","text":"A StorageClass provides a way for administrators to describe the \"classes\" of storage they offer. Different classes might map to quality-of-service levels, or to backup policies, or to arbitrary policies determined by the cluster administrators. Kubernetes itself is un-opinionated about what classes represent. This concept is sometimes called \"profiles\" in other storage systems. An example of a storage class is below: apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: standard provisioner: kubernetes.io/aws-ebs parameters: type: gp2 reclaimPolicy: Retain allowVolumeExpansion: true mountOptions: - debug volumeBindingMode: Immediate Key declarations are: provisioner : Determine which volume plugin to use. This usually matches to a cloud provider, and a specific storage service that it offers. In this example AWS Elastic Block Store parameters : Describe characteristics of this storage class in context of the underlying provisioner. In this example, the type is gp2 which, in AWS lingo means General Purpose SSD. Other types include IO1 (Provisioned IOPS), ST1 (Throughput Optimised) and STC (Cold Storage). A storageclass object by itself defines what storage is provisioned when it is invoked . By itself, it does nothing. A persistentvolume object can be used to request storage from a storageclass and is typically part of a pod manifest apiVersion: v1 kind: PersistentVolume metadata: name: pv0003 spec: capacity: storage: 5Gi volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Recycle storageClassName: standard","title":"Understand storage classes, persistent volumes"},{"location":"revision-topics/04-storage/#understand-volume-mode-access-modes-and-reclaim-policies-for-volumes","text":"","title":"Understand volume mode, access modes and reclaim policies for volumes"},{"location":"revision-topics/04-storage/#volume-modes","text":"Only two exist: block - Mounted to a pod as a raw block device without a filesystem. The Pod / application needs to understand how to deal with raw block devices. Presenting it in this way can yield better performance, at the expense of complexity. filesystem - Mounted inside a pods' filesystem inside a directory. If the volume is backed by a block device with no filesystem, Kubernetes will create one. Compared to block devices, this method offers the highest compatibility, at the expense of performance.","title":"Volume modes"},{"location":"revision-topics/04-storage/#access-modes","text":"Three options exist: ReadWriteOnce \u2013 The volume can be mounted as read-write by a single node ReadOnlyMany \u2013 The volume can be mounted read-only by many nodes ReadWriteMany \u2013 The volume can be mounted as read-write by many nodes","title":"Access Modes"},{"location":"revision-topics/04-storage/#understand-persistent-volume-claims-primitive","text":"A PersistentVolume can be thought of as storage provisioned by an administrator. Think of this as pre-allocation A PersistentVolumeClaim can be thought of as storage requested by a user/workload. When the user creates PVC to request storage, Kubernetes will try to match that PVC with a pre-allocated PV. If a match can be found, the PVC will be bound to the PV, and the user will start to use that pre-allocated piece of storage.","title":"Understand persistent volume claims primitive"},{"location":"revision-topics/04-storage/#know-how-to-configure-applications-with-persistent-storage","text":"When not using storageclasses there are a number of steps: Create the PV: apiVersion: v1 kind: PersistentVolume metadata: name: task-pv-volume labels: type: local spec: storageClassName: manual capacity: storage: 10Gi accessModes: - ReadWriteOnce hostPath: path: \"/mnt/data\" Create the workload to leverage this: apiVersion: v1 kind: PersistentVolumeClaim metadata: name: task-pv-claim spec: storageClassName: manual accessModes: - ReadWriteOnce resources: requests: storage: 3Gi --- apiVersion: v1 kind: Pod metadata: name: task-pv-pod spec: volumes: - name: task-pv-storage persistentVolumeClaim: claimName: task-pv-claim containers: - name: task-pv-container image: nginx ports: - containerPort: 80 name: \"http-server\" volumeMounts: - mountPath: \"/usr/share/nginx/html\" name: task-pv-storage If leveraging a storageclass, a persistentvolume object is not required, we just need a persistentvolumeclaim apiVersion: v1 kind: PersistentVolumeClaim metadata: name: task-pv-claim spec: storageClassName: myStorageClass accessModes: - ReadWriteOnce resources: requests: storage: 3Gi --- apiVersion: v1 kind: Pod metadata: name: task-pv-pod spec: volumes: - name: task-pv-storage persistentVolumeClaim: claimName: task-pv-claim containers: - name: task-pv-container image: nginx ports: - containerPort: 80 name: \"http-server\" volumeMounts: - mountPath: \"/usr/share/nginx/html\" name: task-pv-storage","title":"Know how to configure applications with persistent storage"},{"location":"revision-topics/05-troubleshooting/","text":"Troubleshooting Evaluate cluster and node logging Master Node(s) ETCD Usually, most etcd implementations also include etcdctl, which can aid in monitoring the state of the cluster. If you\u2019re unsure where to find it, execute the following: find / -name etcdctl Leveraging this tool to check the cluster status: etcdctl --write-out=table --endpoints=$ENDPOINTS endpoint status +------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | https://127.0.0.1:2379 | 4e30a295f2c3c1a4 | 3.5.0 | 8.1 MB | true | false | 3 | 7903 | 7903 | | +------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ The cluster this was executed on has only one master node, hence only one result from the script. You will normally receive a response for each etcd member in the cluster. Alternatively, leverage kubectl get componentstatuses: kubectl get componentstatuses #ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok etcd-1 Healthy {\"health\":\"true\"} etcd-0 Healthy {\"health\":\"true\"} Etcd may also be running as a Pod: kubectl logs etcd-ubuntu -n kube-system Kube-apiserver This is dependent on the environment for which the Kubernetes platform has been installed on. For systemd based systems: journalctl -u kube-apiserver Or cat /var/log/kube-apiserver.log Or for instances where Kube-API server is running as a static pod: kubectl logs kube-apiserver-k8s-master-03 -n kube-system Kube-Scheduler For systemd-based systems journalctl -u kube-scheduler Or cat /var/log/kube-scheduler.log Or for instances where Kube-Scheduler is running as a static pod: kubectl logs kube-scheduler-k8s-master-03 -n kube-system Kube-Controller-Manager For systemd-based systems journalctl -u kube-controller-manager Or cat /var/log/kube-controller-manager.log Or for instances where Kube-controller manager is running as a static pod: kubectl logs kube-controller-manager-k8s-master-03 -n kube-system Worker Node(s) CNI Obviously this is dependent on the CNI in use for the cluster you\u2019re working on. However, using Flannel as an example: journalctl -u flanneld If running as a pod, however: Kubectl logs --namespace kube-system <POD-ID> -c kube-flannel kubectl logs --namespace kube-system weave-net-pwjkj -c weave Kube-Proxy For systemd-based systems journalctl -u kube-proxy Or cat /var/log/kube-proxy.log Or for instances where Kube-proxy manager is running as a static pod: kubectl logs kube-proxy -n kube-system Kubelet journalctl -u kubelet Or cat /var/log/kubelet.log Container Runtime Similarly to the CNI, this depends on which container runtime has been deployed, but using Docker as an example: For systemd-based systems: journalctl -u docker.service Or cat /var/log/docker.log Hint : list the contents of etc/systemd/system if it\u2019s a systemd-based service (containerd.service may be here) Cluster Logging At a cluster level, kubectl get events provides a good overview. Understand how to monitor applications This section is a bit open-ended as it highly depends on what you have deployed and the topology of an application. Typically, however, we have an application that runs as a number of inter-connected microservices , consequently we monitor our applications by monitoring the underlying objects that comprise it, such as: Pods Deployments Services etc Manage container stdout & stderr logs Kubernetes handles and redirects any output generated from a containers stdout and stderr streams. These get directed through a logging driver which influences where to store these logs. Different implementations of Docker differ in exact implementation (such as RHEL's flavor of Docker) but commonly, these drivers will write to a file in json format: root@ubuntu:~# docker info | grep \"Logging Driver\" Logging Driver: json-file The location for these logs is typically /var/log/containers but can be tweaked. Additionally, these contain symlinks: root@ubuntu:~# ls -la /var/log/containers/ total 44 drwxr-xr-x 2 root root 4096 Feb 8 19:18 . drwxrwxr-x 11 root syslog 4096 Feb 12 00:00 .. lrwxrwxrwx 1 root root 100 Feb 8 19:17 coredns-74ff55c5b-j4trd_kube-system_coredns-5d65324791ffcdf45d3552d875c6834f9a305c5be84b18745cb1657f784e5dd0.log -> /var/log/pods/kube-system_coredns-74ff55c5b-j4trd_4afbef57-5592-4edb-96af-9d17f595d160/coredns/0.log lrwxrwxrwx 1 root root 100 Feb 8 19:17 coredns-74ff55c5b-wrgkr_kube-system_coredns-b2fcfa679e9725dbe601bc1a0f218121a9c44b91d7300bbb57039a4edd219991.log -> /var/log/pods/kube-system_coredns-74ff55c5b-wrgkr_b64ac6d5-654b-4194-b6a8-5f8aa4c3cbe2/coredns/0.log lrwxrwxrwx 1 root root 81 Feb 8 19:16 etcd-ubuntu_kube-system_etcd-fcc5bc99932f380781776baa125b6f3be035e18fcec520afb827102e2afce1cd.log -> /var/log/pods/kube-system_etcd-ubuntu_f608198a8b73b3cf090bd15e2823df04/etcd/0.log lrwxrwxrwx 1 root root 101 Feb 8 19:16 kube-apiserver-ubuntu_kube-system_kube-apiserver-a264bbd54b7f23c8d424b0b368a48fdd1c5dcecc72fca95a460c146b2b5d85f5.log -> /var/log/pods/kube-system_kube-apiserver-ubuntu_212641053a16fa2bb404ccde20f6eaf0/kube-apiserver/0.log lrwxrwxrwx 1 root root 119 Feb 8 19:17 kube-controller-manager-ubuntu_kube-system_kube-controller-manager-a4ef7fe2b52272ea77f8de2da0989a9bcee757ae778fc08f1786b26b45bf13e1.log -> /var/log/pods/kube-system_kube-controller-manager-ubuntu_7bbe7d37f1b2c7586237165580c2f5c3/kube-controller-manager/0.log lrwxrwxrwx 1 root root 102 Feb 8 19:05 kube-flannel-ds-rfsfs_kube-system_install-cni-f8762e22fbf17925432682bdb1259a066208c62fa695d09cd6ee9b0cef3d36ba.log -> /var/log/pods/kube-system_kube-flannel-ds-rfsfs_2892d4e3-e326-4b4b-90c0-396fb80863ca/install-cni/0.log lrwxrwxrwx 1 root root 103 Feb 8 19:05 kube-flannel-ds-rfsfs_kube-system_kube-flannel-f96e019717814d7360e1aacd275cac121c13e0ee94cc5c93dcb35365608e6f83.log -> /var/log/pods/kube-system_kube-flannel-ds-rfsfs_2892d4e3-e326-4b4b-90c0-396fb80863ca/kube-flannel/0.log lrwxrwxrwx 1 root root 96 Feb 8 19:17 kube-proxy-l52f9_kube-system_kube-proxy-bfe08cb8663b46551e8608c094194ec61d03edfa7d25a6f414c07ed6563ada89.log -> /var/log/pods/kube-system_kube-proxy-l52f9_d2b73ed1-5df4-4a18-9595-20798db4f110/kube-proxy/0.log lrwxrwxrwx 1 root root 101 Feb 8 19:17 kube-scheduler-ubuntu_kube-system_kube-scheduler-4a695e53684f4591ec9385d6944f7841c0329aa49be220e5af6304da281cb41a.log -> /var/log/pods/kube-system_kube-scheduler-ubuntu_69cd289b4ed80ced4f95a59ff60fa102/kube-scheduler/0.log Troubleshoot application failure This is a somewhat ambitious topic to cover as how we approach troubleshooting application failures varies by the architecture of that application, which resources/API objects we're leveraging, if the application contains logs. However, good starting points would include running things like: kubectl describe <object> kubectl logs <podname> kubectl get events Troubleshoot cluster component failure Covered in \"Evaluate cluster and node logging\" Troubleshoot networking DNS Resolution Pods and Services will automatically have a DNS record registered against coredns in the cluster, aka \"A\" records for IPv4 and \"AAAA\" for IPv6. The format of which is: pod-ip-address.my-namespace.pod.cluster-domain.example my-svc-name.my-namespace.svc.cluster-domain.example Pod DNS records resolve to a single entity, even if the Pod contains multiple containers as they share the same networking space. Service DNS records resolve to the respective service object. Pods will automatically have their DNS resolution configured based on coredns settings. This can be validated by opening a shell to the pod and inspecting /etc/resolv.conf: > kubectl exec -it web-server sh kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead. / # cat /etc/resolv.conf nameserver 10.43.0.10 search default.svc.cluster.local svc.cluster.local cluster.local eu-central-1.compute.internal options ndots:5 10.43.0.10 being the coredns service object: > kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 16d To test resolution, we can run a pod with nslookup to test. For the pod below: > kubectl get po -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES web-server 1/1 Running 0 2d20h 10.42.1.31 ip-172-31-36-67 <none> <none> Knowing the format of the A record: pod-ip-address.my-namespace.pod.cluster-domain.example We should be able to resolve 10-42-1-31.default.pod.cluster.local . Tip : To determine the cluster domain, inspect the coredns configmap. Below indicating cluster.local . > kubectl get cm coredns -n kube-system -o yaml apiVersion: v1 data: Corefile: | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { Create a Pod with the tools required: kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml Test lookup: kubectl exec -i -t dnsutils -- nslookup 10-42-1-31.default.pod.cluster.local > kubectl exec -i -t dnsutils -- nslookup 10-42-1-31.default.pod.cluster.local Server: 10.43.0.10 Address: 10.43.0.10#53 Name: 10-42-1-31.default.pod.cluster.local Address: 10.42.1.31 Similarly, for a service, in this case a service called nginx-service that resides in the default namespace: > kubectl exec -i -t dnsutils -- nslookup nginx-service.default.svc.cluster.local Server: 10.43.0.10 Address: 10.43.0.10#53 Name: nginx-service.default.svc.cluster.local Address: 10.43.0.223 > kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx-service ClusterIP 10.43.0.223 <none> 80/TCP 9m15s CNI Issues Mainly covered earlier in acquiring logs for the CNI. However, one issue that might occur is when a CNI is incorrectly, or not initialised. This may cause workloads to enter a pending status: kubectl get po -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx 0/1 Pending 0 57s <none> <none> <none> <none> kubectl describe <pod> can help identify issues with assigning IP addresses to nodes from the CNI Port Checking Similarly, with leveraging nslookup to validate DNS resolution in our cluster, we can lean on other tools to perform other diagnostic. All we need is a pod that has a utility like netcat , telnet etc.","title":"Part Five - Troubleshooting"},{"location":"revision-topics/05-troubleshooting/#troubleshooting","text":"","title":"Troubleshooting"},{"location":"revision-topics/05-troubleshooting/#evaluate-cluster-and-node-logging","text":"","title":"Evaluate cluster and node logging"},{"location":"revision-topics/05-troubleshooting/#master-nodes","text":"","title":"Master Node(s)"},{"location":"revision-topics/05-troubleshooting/#etcd","text":"Usually, most etcd implementations also include etcdctl, which can aid in monitoring the state of the cluster. If you\u2019re unsure where to find it, execute the following: find / -name etcdctl Leveraging this tool to check the cluster status: etcdctl --write-out=table --endpoints=$ENDPOINTS endpoint status +------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | https://127.0.0.1:2379 | 4e30a295f2c3c1a4 | 3.5.0 | 8.1 MB | true | false | 3 | 7903 | 7903 | | +------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ The cluster this was executed on has only one master node, hence only one result from the script. You will normally receive a response for each etcd member in the cluster. Alternatively, leverage kubectl get componentstatuses: kubectl get componentstatuses #ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok etcd-1 Healthy {\"health\":\"true\"} etcd-0 Healthy {\"health\":\"true\"} Etcd may also be running as a Pod: kubectl logs etcd-ubuntu -n kube-system","title":"ETCD"},{"location":"revision-topics/05-troubleshooting/#kube-apiserver","text":"This is dependent on the environment for which the Kubernetes platform has been installed on. For systemd based systems: journalctl -u kube-apiserver Or cat /var/log/kube-apiserver.log Or for instances where Kube-API server is running as a static pod: kubectl logs kube-apiserver-k8s-master-03 -n kube-system","title":"Kube-apiserver"},{"location":"revision-topics/05-troubleshooting/#kube-scheduler","text":"For systemd-based systems journalctl -u kube-scheduler Or cat /var/log/kube-scheduler.log Or for instances where Kube-Scheduler is running as a static pod: kubectl logs kube-scheduler-k8s-master-03 -n kube-system","title":"Kube-Scheduler"},{"location":"revision-topics/05-troubleshooting/#kube-controller-manager","text":"For systemd-based systems journalctl -u kube-controller-manager Or cat /var/log/kube-controller-manager.log Or for instances where Kube-controller manager is running as a static pod: kubectl logs kube-controller-manager-k8s-master-03 -n kube-system","title":"Kube-Controller-Manager"},{"location":"revision-topics/05-troubleshooting/#worker-nodes","text":"","title":"Worker Node(s)"},{"location":"revision-topics/05-troubleshooting/#cni","text":"Obviously this is dependent on the CNI in use for the cluster you\u2019re working on. However, using Flannel as an example: journalctl -u flanneld If running as a pod, however: Kubectl logs --namespace kube-system <POD-ID> -c kube-flannel kubectl logs --namespace kube-system weave-net-pwjkj -c weave","title":"CNI"},{"location":"revision-topics/05-troubleshooting/#kube-proxy","text":"For systemd-based systems journalctl -u kube-proxy Or cat /var/log/kube-proxy.log Or for instances where Kube-proxy manager is running as a static pod: kubectl logs kube-proxy -n kube-system","title":"Kube-Proxy"},{"location":"revision-topics/05-troubleshooting/#kubelet","text":"journalctl -u kubelet Or cat /var/log/kubelet.log","title":"Kubelet"},{"location":"revision-topics/05-troubleshooting/#container-runtime","text":"Similarly to the CNI, this depends on which container runtime has been deployed, but using Docker as an example: For systemd-based systems: journalctl -u docker.service Or cat /var/log/docker.log Hint : list the contents of etc/systemd/system if it\u2019s a systemd-based service (containerd.service may be here)","title":"Container Runtime"},{"location":"revision-topics/05-troubleshooting/#cluster-logging","text":"At a cluster level, kubectl get events provides a good overview.","title":"Cluster Logging"},{"location":"revision-topics/05-troubleshooting/#understand-how-to-monitor-applications","text":"This section is a bit open-ended as it highly depends on what you have deployed and the topology of an application. Typically, however, we have an application that runs as a number of inter-connected microservices , consequently we monitor our applications by monitoring the underlying objects that comprise it, such as: Pods Deployments Services etc","title":"Understand how to monitor applications"},{"location":"revision-topics/05-troubleshooting/#manage-container-stdout-stderr-logs","text":"Kubernetes handles and redirects any output generated from a containers stdout and stderr streams. These get directed through a logging driver which influences where to store these logs. Different implementations of Docker differ in exact implementation (such as RHEL's flavor of Docker) but commonly, these drivers will write to a file in json format: root@ubuntu:~# docker info | grep \"Logging Driver\" Logging Driver: json-file The location for these logs is typically /var/log/containers but can be tweaked. Additionally, these contain symlinks: root@ubuntu:~# ls -la /var/log/containers/ total 44 drwxr-xr-x 2 root root 4096 Feb 8 19:18 . drwxrwxr-x 11 root syslog 4096 Feb 12 00:00 .. lrwxrwxrwx 1 root root 100 Feb 8 19:17 coredns-74ff55c5b-j4trd_kube-system_coredns-5d65324791ffcdf45d3552d875c6834f9a305c5be84b18745cb1657f784e5dd0.log -> /var/log/pods/kube-system_coredns-74ff55c5b-j4trd_4afbef57-5592-4edb-96af-9d17f595d160/coredns/0.log lrwxrwxrwx 1 root root 100 Feb 8 19:17 coredns-74ff55c5b-wrgkr_kube-system_coredns-b2fcfa679e9725dbe601bc1a0f218121a9c44b91d7300bbb57039a4edd219991.log -> /var/log/pods/kube-system_coredns-74ff55c5b-wrgkr_b64ac6d5-654b-4194-b6a8-5f8aa4c3cbe2/coredns/0.log lrwxrwxrwx 1 root root 81 Feb 8 19:16 etcd-ubuntu_kube-system_etcd-fcc5bc99932f380781776baa125b6f3be035e18fcec520afb827102e2afce1cd.log -> /var/log/pods/kube-system_etcd-ubuntu_f608198a8b73b3cf090bd15e2823df04/etcd/0.log lrwxrwxrwx 1 root root 101 Feb 8 19:16 kube-apiserver-ubuntu_kube-system_kube-apiserver-a264bbd54b7f23c8d424b0b368a48fdd1c5dcecc72fca95a460c146b2b5d85f5.log -> /var/log/pods/kube-system_kube-apiserver-ubuntu_212641053a16fa2bb404ccde20f6eaf0/kube-apiserver/0.log lrwxrwxrwx 1 root root 119 Feb 8 19:17 kube-controller-manager-ubuntu_kube-system_kube-controller-manager-a4ef7fe2b52272ea77f8de2da0989a9bcee757ae778fc08f1786b26b45bf13e1.log -> /var/log/pods/kube-system_kube-controller-manager-ubuntu_7bbe7d37f1b2c7586237165580c2f5c3/kube-controller-manager/0.log lrwxrwxrwx 1 root root 102 Feb 8 19:05 kube-flannel-ds-rfsfs_kube-system_install-cni-f8762e22fbf17925432682bdb1259a066208c62fa695d09cd6ee9b0cef3d36ba.log -> /var/log/pods/kube-system_kube-flannel-ds-rfsfs_2892d4e3-e326-4b4b-90c0-396fb80863ca/install-cni/0.log lrwxrwxrwx 1 root root 103 Feb 8 19:05 kube-flannel-ds-rfsfs_kube-system_kube-flannel-f96e019717814d7360e1aacd275cac121c13e0ee94cc5c93dcb35365608e6f83.log -> /var/log/pods/kube-system_kube-flannel-ds-rfsfs_2892d4e3-e326-4b4b-90c0-396fb80863ca/kube-flannel/0.log lrwxrwxrwx 1 root root 96 Feb 8 19:17 kube-proxy-l52f9_kube-system_kube-proxy-bfe08cb8663b46551e8608c094194ec61d03edfa7d25a6f414c07ed6563ada89.log -> /var/log/pods/kube-system_kube-proxy-l52f9_d2b73ed1-5df4-4a18-9595-20798db4f110/kube-proxy/0.log lrwxrwxrwx 1 root root 101 Feb 8 19:17 kube-scheduler-ubuntu_kube-system_kube-scheduler-4a695e53684f4591ec9385d6944f7841c0329aa49be220e5af6304da281cb41a.log -> /var/log/pods/kube-system_kube-scheduler-ubuntu_69cd289b4ed80ced4f95a59ff60fa102/kube-scheduler/0.log","title":"Manage container stdout &amp; stderr logs"},{"location":"revision-topics/05-troubleshooting/#troubleshoot-application-failure","text":"This is a somewhat ambitious topic to cover as how we approach troubleshooting application failures varies by the architecture of that application, which resources/API objects we're leveraging, if the application contains logs. However, good starting points would include running things like: kubectl describe <object> kubectl logs <podname> kubectl get events","title":"Troubleshoot application failure"},{"location":"revision-topics/05-troubleshooting/#troubleshoot-cluster-component-failure","text":"Covered in \"Evaluate cluster and node logging\"","title":"Troubleshoot cluster component failure"},{"location":"revision-topics/05-troubleshooting/#troubleshoot-networking","text":"","title":"Troubleshoot networking"},{"location":"revision-topics/05-troubleshooting/#dns-resolution","text":"Pods and Services will automatically have a DNS record registered against coredns in the cluster, aka \"A\" records for IPv4 and \"AAAA\" for IPv6. The format of which is: pod-ip-address.my-namespace.pod.cluster-domain.example my-svc-name.my-namespace.svc.cluster-domain.example Pod DNS records resolve to a single entity, even if the Pod contains multiple containers as they share the same networking space. Service DNS records resolve to the respective service object. Pods will automatically have their DNS resolution configured based on coredns settings. This can be validated by opening a shell to the pod and inspecting /etc/resolv.conf: > kubectl exec -it web-server sh kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead. / # cat /etc/resolv.conf nameserver 10.43.0.10 search default.svc.cluster.local svc.cluster.local cluster.local eu-central-1.compute.internal options ndots:5 10.43.0.10 being the coredns service object: > kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 16d To test resolution, we can run a pod with nslookup to test. For the pod below: > kubectl get po -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES web-server 1/1 Running 0 2d20h 10.42.1.31 ip-172-31-36-67 <none> <none> Knowing the format of the A record: pod-ip-address.my-namespace.pod.cluster-domain.example We should be able to resolve 10-42-1-31.default.pod.cluster.local . Tip : To determine the cluster domain, inspect the coredns configmap. Below indicating cluster.local . > kubectl get cm coredns -n kube-system -o yaml apiVersion: v1 data: Corefile: | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { Create a Pod with the tools required: kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml Test lookup: kubectl exec -i -t dnsutils -- nslookup 10-42-1-31.default.pod.cluster.local > kubectl exec -i -t dnsutils -- nslookup 10-42-1-31.default.pod.cluster.local Server: 10.43.0.10 Address: 10.43.0.10#53 Name: 10-42-1-31.default.pod.cluster.local Address: 10.42.1.31 Similarly, for a service, in this case a service called nginx-service that resides in the default namespace: > kubectl exec -i -t dnsutils -- nslookup nginx-service.default.svc.cluster.local Server: 10.43.0.10 Address: 10.43.0.10#53 Name: nginx-service.default.svc.cluster.local Address: 10.43.0.223 > kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx-service ClusterIP 10.43.0.223 <none> 80/TCP 9m15s","title":"DNS Resolution"},{"location":"revision-topics/05-troubleshooting/#cni-issues","text":"Mainly covered earlier in acquiring logs for the CNI. However, one issue that might occur is when a CNI is incorrectly, or not initialised. This may cause workloads to enter a pending status: kubectl get po -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx 0/1 Pending 0 57s <none> <none> <none> <none> kubectl describe <pod> can help identify issues with assigning IP addresses to nodes from the CNI","title":"CNI Issues"},{"location":"revision-topics/05-troubleshooting/#port-checking","text":"Similarly, with leveraging nslookup to validate DNS resolution in our cluster, we can lean on other tools to perform other diagnostic. All we need is a pod that has a utility like netcat , telnet etc.","title":"Port Checking"}]}
\ No newline at end of file
+{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"CKA-Study Guide The following guide is based on the Certified Kubernetes Administrator Exam Curriculum 1.29 and consists of the following sections: Revision Topics Notes from revising the topics listed in the curriculum, grouped to correlate to the various sections listed in the curriculum. Lab Guide With Example Answers To complement the revision topics, these lab guides are designed to exercise the topics learnt from the corresponding revision topics sections. Resources / References http://www.kubernet.io/ (Currently offline) https://github.com/kelseyhightower/kubernetes-the-hard-way Certified Kubernetes Administrator Course on A Cloud Guru https://kubernetes.io/docs/ https://kubernetes.io/docs/tasks/ Lab Environments Minikube Setup WSL Docker Kubernetes on the Windows Desktop Contact Twitter/X = @VT_UK","title":"Home"},{"location":"#cka-study-guide","text":"The following guide is based on the Certified Kubernetes Administrator Exam Curriculum 1.29 and consists of the following sections:","title":"CKA-Study Guide"},{"location":"#revision-topics","text":"Notes from revising the topics listed in the curriculum, grouped to correlate to the various sections listed in the curriculum.","title":"Revision Topics"},{"location":"#lab-guide-with-example-answers","text":"To complement the revision topics, these lab guides are designed to exercise the topics learnt from the corresponding revision topics sections.","title":"Lab Guide With Example Answers"},{"location":"#resources-references","text":"http://www.kubernet.io/ (Currently offline) https://github.com/kelseyhightower/kubernetes-the-hard-way Certified Kubernetes Administrator Course on A Cloud Guru https://kubernetes.io/docs/ https://kubernetes.io/docs/tasks/","title":"Resources / References"},{"location":"#lab-environments","text":"Minikube Setup WSL Docker Kubernetes on the Windows Desktop","title":"Lab Environments"},{"location":"#contact","text":"Twitter/X = @VT_UK","title":"Contact"},{"location":"lab-guide/00-general-advice/","text":"General Advice Imperative vs Declarative The Lab guide will present both imperative and declarative answers to questions where possible. Due to time constraints in the exam, using imperative commands may be more efficient. Outside of the exam, declarative management is the preferred approach. Use the documentation At time of writing, Kubernetes Documentation , Kubernetes GitHub and Kubernetes Blog are permitted to be used during the exam as per the current guidelines . If you need to get boilerplate YAML manifests to address an exam question, feel free to get them from here. The key is to know what to search for. For example, if asked to create and apply a service object, search for Service YAML in the Kubernetes docs and modify an example to suit the question objectives. Caution on documentation When using the Kubernetes.io docs, be careful of the search results - some will point to resources outside of kubernetes.io and the list of accepted resource for the exam. Should you navigate to these your exam may be cancelled. Sanity check the URL the search result is pointing to prior to procedding. Don't write YAML manifests from scratch In addition to the above example, leverage the combination of --dry-run and -o to generate YAML that can be modified. kubectl run nginx --image=nginx --dry-run=client -o yaml > pod.yaml Which will generate: apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: nginx name: nginx spec: containers: - image: nginx name: nginx resources: {} dnsPolicy: ClusterFirst restartPolicy: Always status: {} --dry-run won't commit the command -o yaml specifies YAML as the output > pod.yaml redirects the output to a file Become BFF's with kubectl Effective use of kubectl will be extremely important to pass the exam. Become very comfortable with it. Unsure of an objects spec, type, options? kubectl explain <resourcetype>","title":"Part Zero - General Advice"},{"location":"lab-guide/00-general-advice/#general-advice","text":"","title":"General Advice"},{"location":"lab-guide/00-general-advice/#imperative-vs-declarative","text":"The Lab guide will present both imperative and declarative answers to questions where possible. Due to time constraints in the exam, using imperative commands may be more efficient. Outside of the exam, declarative management is the preferred approach.","title":"Imperative vs Declarative"},{"location":"lab-guide/00-general-advice/#use-the-documentation","text":"At time of writing, Kubernetes Documentation , Kubernetes GitHub and Kubernetes Blog are permitted to be used during the exam as per the current guidelines . If you need to get boilerplate YAML manifests to address an exam question, feel free to get them from here. The key is to know what to search for. For example, if asked to create and apply a service object, search for Service YAML in the Kubernetes docs and modify an example to suit the question objectives.","title":"Use the documentation"},{"location":"lab-guide/00-general-advice/#caution-on-documentation","text":"When using the Kubernetes.io docs, be careful of the search results - some will point to resources outside of kubernetes.io and the list of accepted resource for the exam. Should you navigate to these your exam may be cancelled. Sanity check the URL the search result is pointing to prior to procedding.","title":"Caution on documentation"},{"location":"lab-guide/00-general-advice/#dont-write-yaml-manifests-from-scratch","text":"In addition to the above example, leverage the combination of --dry-run and -o to generate YAML that can be modified. kubectl run nginx --image=nginx --dry-run=client -o yaml > pod.yaml Which will generate: apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: nginx name: nginx spec: containers: - image: nginx name: nginx resources: {} dnsPolicy: ClusterFirst restartPolicy: Always status: {} --dry-run won't commit the command -o yaml specifies YAML as the output > pod.yaml redirects the output to a file","title":"Don't write YAML manifests from scratch"},{"location":"lab-guide/00-general-advice/#become-bffs-with-kubectl","text":"Effective use of kubectl will be extremely important to pass the exam. Become very comfortable with it.","title":"Become BFF's with kubectl"},{"location":"lab-guide/00-general-advice/#unsure-of-an-objects-spec-type-options","text":"kubectl explain <resourcetype>","title":"Unsure of an objects spec, type, options?"},{"location":"lab-guide/01-architcture-installation-configuration/","text":"Lab Exercises for Cluster Architecture, Installation and Configuration Exercise 0 - Setup Prepare a cluster (Single node, kubeadm, k3s, etc) Prepare two vanilla VM's (No Kubernetes components installed) with the kubeadm binary installed (feel free to do this beforehand, I doubt it will be a requirement to add apt sources to get it) Open browser tabs to Kubernetes Documentation , Kubernetes GitHub and Kubernetes Blog (these are permitted as per the current guidelines ) Exercise 1 - RBAC A third party application requires access to describe job objects that reside in a namespace called rbac . Perform the following: Create a namespace called rbac Create a service account called job-inspector for the rbac namespace Create a role that has rules to get and list job objects Create a rolebinding that binds the service account job-inspector to the role created in step 3 Prove the job-inspector service account can \"get\" job objects but not deployment objects Answer - Imperative kubectl create namespace rbac kubectl create sa job-inspector -n rbac kubectl create role job-inspector --verb=get --verb=list --resource=jobs -n rbac kubectl create rolebinding permit-job-inspector --role=job-inspector --serviceaccount=rbac:job-inspector -n rbac kubectl --as=system:serviceaccount:rbac:job-inspector auth can-i get job -n rbac kubectl --as=system:serviceaccount:rbac:job-inspector auth can-i get deployment -n rbac Answer - Declarative apiVersion: v1 kind: Namespace metadata: name: rbac --- apiVersion: v1 kind: ServiceAccount metadata: name: job-inspector namespace: rbac --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: job-inspector namespace: rbac rules: - apiGroups: [\"batch\"] resources: [\"jobs\"] verbs: [\"get\", \"list\"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: permit-job-inspector namespace: rbac roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: job-inspector subjects: - kind: ServiceAccount name: job-inspector namespace: rbac Exercise 2 - Use Kubeadm to install a basic cluster On a node, install kubeadm and stand up the control plane, using 10.244.0.0/16 as the pod network CIDR, and kube-flannel.yml as the CNI On a node, install kubeadm and join it to the cluster as a worker node Answer Node 1 Prep kubeadm (as mentioned above, I doubt we will need to do this part in the exam) # Install a container runtime, IE https://github.com/containerd/containerd/blob/main/docs/getting-started.md sudo apt update && apt install containerd -y sudo apt-get update # apt-transport-https may be a dummy package; if so, you can skip that package sudo apt-get install -y apt-transport-https ca-certificates curl gpg # If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below. # sudo mkdir -p -m 755 /etc/apt/keyrings curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg # This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list sudo apt-get update sudo apt-get install -y kubelet kubeadm kubectl sudo apt-mark hold kubelet kubeadm kubectl You may need to execute the following depending on the underlying OS: modprobe br_netfilter echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables echo 1 > /proc/sys/net/ipv4/ip_forward Turn this node into a master sudo kubeadm init --pod-network-cidr=10.244.0.0/16 ... mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config ... kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml ... Note the join command, ie: kubeadm join 172.16.10.210:6443 --token 9tjntl.10plpxqy85g8a0ui \\ --discovery-token-ca-cert-hash sha256:381165c9a9f19a123bd0fee36fe36d15e918062dcc94711ff5b286ee1f86b92b Node 2: Run the join command taken from the previous step kubeadm join 172.16.10.210:6443 --token 9tjntl.10plpxqy85g8a0ui \\ --discovery-token-ca-cert-hash sha256:381165c9a9f19a123bd0fee36fe36d15e918062dcc94711ff5b286ee1f86b92b Validate by running kubectl get no on the master node: kubectl get no NAME STATUS ROLES AGE VERSION ubuntu Ready control-plane,master 9m53s v1.31.3 ubuntu2 Ready <none> 50s v1.31.3 Exercise 3 - Manage a highly-available Kubernetes cluster Using etcdctl , determine the health of the etcd cluster Using etcdctl , identify the list of members On the master node, determine the health of the cluster by probing the API endpoint Answer root@ip-172-31-31-80:~# ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \\ --cacert=/etc/kubernetes/pki/etcd/ca.crt \\ --cert=/etc/kubernetes/pki/etcd/server.crt \\ --key=/etc/kubernetes/pki/etcd/server.key \\ endpoint health https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 6.852757ms root@ip-172-31-31-80:~# ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \\ --cacert=/etc/kubernetes/pki/etcd/ca.crt \\ --cert=/etc/kubernetes/pki/etcd/server.crt \\ --key=/etc/kubernetes/pki/etcd/server.key \\ member list f90d942cc570125d, started, ip-172-31-31-80, https://172.31.31.80:2380, https://172.31.31.80:2379, false curl -k https://localhost:6443/healthz?verbose [+]ping ok [+]log ok [+]etcd ok [+]poststarthook/start-kube-apiserver-admission-initializer ok ... Exercise 4 - Perform a version upgrade on a Kubernetes cluster using Kubeadm Using kubeadm , upgrade a cluster to the lastest version Answer If held, unhold the kubeadm version sudo apt-mark unhold kubeadm Upgrade the kubeadm version: sudo apt-get install --only-upgrade kubeadm plan the upgrade: sudo kubeadm upgrade plan Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': COMPONENT CURRENT AVAILABLE kubelet 1 x v1.29.10 v1.31.3 Upgrade to the latest stable version: COMPONENT CURRENT AVAILABLE kube-apiserver v1.29.10 v1.31.3 kube-controller-manager v1.29.10 v1.31.3 kube-scheduler v1.29.10 v1.31.3 kube-proxy v1.29.10 v1.31.3 CoreDNS 1.7.0 1.11.3 etcd 3.4.9-1 3.5.15-0 Upgrade the cluster kubeadm upgrade apply v1.20.2 Upgrade Kubelet: sudo apt-get install --only-upgrade kubelet Exercise 5 - Implement etcd backup and restore Take a backup of etcd Verify the etcd backup has been successful Restore the backup back to the cluster Answer Take a snapshot of etcd: ETCDCTL_API=3 etcdctl snapshot save snapshot.db --cacert /etc/kubernetes/pki/etcd/server.crt --cert /etc/kubernetes/pki/etcd/ca.crt --key /etc/kubernetes/pki/etcd/ca.key Verify the snapshot: sudo ETCDCTL_API=3 etcdctl --write-out=table snapshot status snapshot.db Perform a restore: ETCDCTL_API=3 etcdctl snapshot restore snapshot.db","title":"Part One - Architecture, Installation and Configuration"},{"location":"lab-guide/01-architcture-installation-configuration/#lab-exercises-for-cluster-architecture-installation-and-configuration","text":"","title":"Lab Exercises for Cluster Architecture, Installation and Configuration"},{"location":"lab-guide/01-architcture-installation-configuration/#exercise-0-setup","text":"Prepare a cluster (Single node, kubeadm, k3s, etc) Prepare two vanilla VM's (No Kubernetes components installed) with the kubeadm binary installed (feel free to do this beforehand, I doubt it will be a requirement to add apt sources to get it) Open browser tabs to Kubernetes Documentation , Kubernetes GitHub and Kubernetes Blog (these are permitted as per the current guidelines )","title":"Exercise 0 - Setup"},{"location":"lab-guide/01-architcture-installation-configuration/#exercise-1-rbac","text":"A third party application requires access to describe job objects that reside in a namespace called rbac . Perform the following: Create a namespace called rbac Create a service account called job-inspector for the rbac namespace Create a role that has rules to get and list job objects Create a rolebinding that binds the service account job-inspector to the role created in step 3 Prove the job-inspector service account can \"get\" job objects but not deployment objects Answer - Imperative kubectl create namespace rbac kubectl create sa job-inspector -n rbac kubectl create role job-inspector --verb=get --verb=list --resource=jobs -n rbac kubectl create rolebinding permit-job-inspector --role=job-inspector --serviceaccount=rbac:job-inspector -n rbac kubectl --as=system:serviceaccount:rbac:job-inspector auth can-i get job -n rbac kubectl --as=system:serviceaccount:rbac:job-inspector auth can-i get deployment -n rbac Answer - Declarative apiVersion: v1 kind: Namespace metadata: name: rbac --- apiVersion: v1 kind: ServiceAccount metadata: name: job-inspector namespace: rbac --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: job-inspector namespace: rbac rules: - apiGroups: [\"batch\"] resources: [\"jobs\"] verbs: [\"get\", \"list\"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: permit-job-inspector namespace: rbac roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: job-inspector subjects: - kind: ServiceAccount name: job-inspector namespace: rbac","title":"Exercise 1 - RBAC"},{"location":"lab-guide/01-architcture-installation-configuration/#exercise-2-use-kubeadm-to-install-a-basic-cluster","text":"On a node, install kubeadm and stand up the control plane, using 10.244.0.0/16 as the pod network CIDR, and kube-flannel.yml as the CNI On a node, install kubeadm and join it to the cluster as a worker node Answer Node 1 Prep kubeadm (as mentioned above, I doubt we will need to do this part in the exam) # Install a container runtime, IE https://github.com/containerd/containerd/blob/main/docs/getting-started.md sudo apt update && apt install containerd -y sudo apt-get update # apt-transport-https may be a dummy package; if so, you can skip that package sudo apt-get install -y apt-transport-https ca-certificates curl gpg # If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below. # sudo mkdir -p -m 755 /etc/apt/keyrings curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg # This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list sudo apt-get update sudo apt-get install -y kubelet kubeadm kubectl sudo apt-mark hold kubelet kubeadm kubectl You may need to execute the following depending on the underlying OS: modprobe br_netfilter echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables echo 1 > /proc/sys/net/ipv4/ip_forward Turn this node into a master sudo kubeadm init --pod-network-cidr=10.244.0.0/16 ... mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config ... kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml ... Note the join command, ie: kubeadm join 172.16.10.210:6443 --token 9tjntl.10plpxqy85g8a0ui \\ --discovery-token-ca-cert-hash sha256:381165c9a9f19a123bd0fee36fe36d15e918062dcc94711ff5b286ee1f86b92b Node 2: Run the join command taken from the previous step kubeadm join 172.16.10.210:6443 --token 9tjntl.10plpxqy85g8a0ui \\ --discovery-token-ca-cert-hash sha256:381165c9a9f19a123bd0fee36fe36d15e918062dcc94711ff5b286ee1f86b92b Validate by running kubectl get no on the master node: kubectl get no NAME STATUS ROLES AGE VERSION ubuntu Ready control-plane,master 9m53s v1.31.3 ubuntu2 Ready <none> 50s v1.31.3","title":"Exercise 2 - Use Kubeadm to install a basic cluster"},{"location":"lab-guide/01-architcture-installation-configuration/#exercise-3-manage-a-highly-available-kubernetes-cluster","text":"Using etcdctl , determine the health of the etcd cluster Using etcdctl , identify the list of members On the master node, determine the health of the cluster by probing the API endpoint Answer root@ip-172-31-31-80:~# ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \\ --cacert=/etc/kubernetes/pki/etcd/ca.crt \\ --cert=/etc/kubernetes/pki/etcd/server.crt \\ --key=/etc/kubernetes/pki/etcd/server.key \\ endpoint health https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 6.852757ms root@ip-172-31-31-80:~# ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \\ --cacert=/etc/kubernetes/pki/etcd/ca.crt \\ --cert=/etc/kubernetes/pki/etcd/server.crt \\ --key=/etc/kubernetes/pki/etcd/server.key \\ member list f90d942cc570125d, started, ip-172-31-31-80, https://172.31.31.80:2380, https://172.31.31.80:2379, false curl -k https://localhost:6443/healthz?verbose [+]ping ok [+]log ok [+]etcd ok [+]poststarthook/start-kube-apiserver-admission-initializer ok ...","title":"Exercise 3 - Manage a highly-available Kubernetes cluster"},{"location":"lab-guide/01-architcture-installation-configuration/#exercise-4-perform-a-version-upgrade-on-a-kubernetes-cluster-using-kubeadm","text":"Using kubeadm , upgrade a cluster to the lastest version Answer If held, unhold the kubeadm version sudo apt-mark unhold kubeadm Upgrade the kubeadm version: sudo apt-get install --only-upgrade kubeadm plan the upgrade: sudo kubeadm upgrade plan Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': COMPONENT CURRENT AVAILABLE kubelet 1 x v1.29.10 v1.31.3 Upgrade to the latest stable version: COMPONENT CURRENT AVAILABLE kube-apiserver v1.29.10 v1.31.3 kube-controller-manager v1.29.10 v1.31.3 kube-scheduler v1.29.10 v1.31.3 kube-proxy v1.29.10 v1.31.3 CoreDNS 1.7.0 1.11.3 etcd 3.4.9-1 3.5.15-0 Upgrade the cluster kubeadm upgrade apply v1.20.2 Upgrade Kubelet: sudo apt-get install --only-upgrade kubelet","title":"Exercise 4 - Perform a version upgrade on a Kubernetes cluster using Kubeadm"},{"location":"lab-guide/01-architcture-installation-configuration/#exercise-5-implement-etcd-backup-and-restore","text":"Take a backup of etcd Verify the etcd backup has been successful Restore the backup back to the cluster Answer Take a snapshot of etcd: ETCDCTL_API=3 etcdctl snapshot save snapshot.db --cacert /etc/kubernetes/pki/etcd/server.crt --cert /etc/kubernetes/pki/etcd/ca.crt --key /etc/kubernetes/pki/etcd/ca.key Verify the snapshot: sudo ETCDCTL_API=3 etcdctl --write-out=table snapshot status snapshot.db Perform a restore: ETCDCTL_API=3 etcdctl snapshot restore snapshot.db","title":"Exercise 5 - Implement etcd backup and restore"},{"location":"lab-guide/02-workloads-and-scheduling/","text":"Lab Exercises for Workloads & Scheduling Exercise 0 - Setup Prepare a cluster (Single node, kubeadm, k3s, etc) Open browser tabs to Kubernetes Documentation , Kubernetes GitHub and Kubernetes Blog (these are permitted as per the current guidelines ) Exercise 1 - Understand deployments and how to perform rolling update and rollbacks Create a deployment object consisting of 6 pods, each containing a single nginx:1.19.5 container Identify the update strategy adopted by this deployment Modify the update strategy so maxSurge is equal to 50% and maxUnavailable is equal to 50% Perform a rolling update to this deployment so that the image gets updated to nginx1.19.6 Undo the most recent change Answer - Imperative kubectl create deployment nginx --image=nginx:1.19.5 --replicas=6 kubectl describe deployment nginx | grep StrategyType kubectl patch deployment nginx -p \"{\\\"spec\\\": {\\\"strategy\\\": {\\\"rollingUpdate\\\": { \\\"maxSurge\\\":\\\"50%\\\"}}}}\" --record=true kubectl patch deployment nginx -p \"{\\\"spec\\\": {\\\"strategy\\\": {\\\"rollingUpdate\\\": { \\\"maxUnavailable\\\":\\\"50%\\\"}}}}\" --record=true kubectl set image deployment nginx nginx=nginx:1.19.6 --record=true kubectl rollout undo deployment/nginx Answer - Declarative apiVersion: apps/v1 kind: Deployment metadata: name: nginx labels: app: nginx spec: replicas: 6 strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.19.5 ports: - containerPort: 80 kubectl describe deployment nginx (unless specified, will be listed as \"UpdateStrategy\") patch.yaml contents: apiVersion: apps/v1 kind: Deployment metadata: name: nginx spec: strategy: rollingUpdate: maxSurge: 50% maxUnavailable: 50% template: spec: containers: - name: nginx image: nginx:1.19.6 kubectl patch deployment nginx --patch-file=patch.yaml kubectl rollout undo deployment/nginx Exercise 2 - Use ConfigMaps to configure applications Create a configmap named mycm that has the following key=value pair key = owner value = yourname Create a pod of your choice, such as nginx . Configure this Pod so that the underlying container has the environent varibale OWNER set to the value of this configmap Answer Create configmap: kubectl create configmap mycm --from-literal=owner=david Define Pod: apiVersion: v1 kind: Pod metadata: name: nginx-configmap spec: containers: - name: nginx-configmap image: nginx command: [ \"/bin/sh\", \"-c\", \"env\" ] env: # Define the environment variable - name: OWNER valueFrom: configMapKeyRef: # The ConfigMap containing the value you want to assign to SPECIAL_LEVEL_KEY name: mycm # Specify the key associated with the value key: owner Validate: kubectl logs nginx-configmap | grep OWNER OWNER=david Exercise 3 - Use Secrets to configure applications Create a secret named mysecret that has the following key=value pair dbuser = MyDatabaseUser dbpassword = MyDatabasePassword Create a pod of your choice, such as nginx . Configure this Pod so that the underlying container has the the following environment variables set: DBUSER from secret key dbuser DBPASS from secret key dbpassword Answer kubectl create secret generic mysecret --from-literal=dbuser=\"MyDatabaseUser\" --from-literal=dbpassword=\"MyDatabasePassword\" Apply the following manifest: apiVersion: v1 kind: Pod metadata: name: nginx-secret spec: containers: - name: nginx-secret image: nginx command: [ \"/bin/sh\", \"-c\", \"env\" ] env: - name: dbuser valueFrom: secretKeyRef: name: mysecret key: dbuser - name: dbpassword valueFrom: secretKeyRef: name: mysecret key: dbpassword restartPolicy: Never Which can be validated with: kubectl logs nginx-secret | grep db dbuser=MyDatabaseUser dbpassword=MyDatabasePassword Exercise 4 - Know how to scale applications Create a deployment object consisting of 3 pods containing a single nginx container Increase this deployment size by adding two additional pods. Decrease this deployment back to the original size of 3 pods Answer - Imperative kubectl create deployment nginx --image=nginx --replicas=3 kubectl scale --replicas=5 deployment nginx kubectl scale --replicas=3 deployment nginx Answer - Declarative Apply initial YAML: apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80 Apply modified YAML: apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: replicas: 5 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1 ports: - containerPort: 80 Apply original YAML: apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80 Excerise 5 - Understand how resource limits can affect Pod scheduling Create a new namespace called \"tenant-b-100mi\" Create a memory limit of 100Mi for this namespace Create a pod with a memory request of 150Mi, ensure the limit has been set by verifying you get a error message. Answer kubectl create ns tenant-b-100mi Deploy the manifest: apiVersion: v1 kind: LimitRange metadata: name: tenant-b-memlimit namespace: tenant-b-100mi spec: limits: - max: memory: 100Mi type: Container Test with deploying: apiVersion: v1 kind: Pod metadata: name: default-mem-demo namespace: tenant-b-100mi spec: containers: - name: default-mem-demo image: nginx resources: requests: memory: 150Mi Which should return: The Pod \"default-mem-demo\" is invalid: spec.containers[0].resources.requests: Invalid value: \"150Mi\": must be less than or equal to memory limit","title":"Part Two - Workloads & Scheduling"},{"location":"lab-guide/02-workloads-and-scheduling/#lab-exercises-for-workloads-scheduling","text":"","title":"Lab Exercises for Workloads &amp; Scheduling"},{"location":"lab-guide/02-workloads-and-scheduling/#exercise-0-setup","text":"Prepare a cluster (Single node, kubeadm, k3s, etc) Open browser tabs to Kubernetes Documentation , Kubernetes GitHub and Kubernetes Blog (these are permitted as per the current guidelines )","title":"Exercise 0 - Setup"},{"location":"lab-guide/02-workloads-and-scheduling/#exercise-1-understand-deployments-and-how-to-perform-rolling-update-and-rollbacks","text":"Create a deployment object consisting of 6 pods, each containing a single nginx:1.19.5 container Identify the update strategy adopted by this deployment Modify the update strategy so maxSurge is equal to 50% and maxUnavailable is equal to 50% Perform a rolling update to this deployment so that the image gets updated to nginx1.19.6 Undo the most recent change Answer - Imperative kubectl create deployment nginx --image=nginx:1.19.5 --replicas=6 kubectl describe deployment nginx | grep StrategyType kubectl patch deployment nginx -p \"{\\\"spec\\\": {\\\"strategy\\\": {\\\"rollingUpdate\\\": { \\\"maxSurge\\\":\\\"50%\\\"}}}}\" --record=true kubectl patch deployment nginx -p \"{\\\"spec\\\": {\\\"strategy\\\": {\\\"rollingUpdate\\\": { \\\"maxUnavailable\\\":\\\"50%\\\"}}}}\" --record=true kubectl set image deployment nginx nginx=nginx:1.19.6 --record=true kubectl rollout undo deployment/nginx Answer - Declarative apiVersion: apps/v1 kind: Deployment metadata: name: nginx labels: app: nginx spec: replicas: 6 strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.19.5 ports: - containerPort: 80 kubectl describe deployment nginx (unless specified, will be listed as \"UpdateStrategy\") patch.yaml contents: apiVersion: apps/v1 kind: Deployment metadata: name: nginx spec: strategy: rollingUpdate: maxSurge: 50% maxUnavailable: 50% template: spec: containers: - name: nginx image: nginx:1.19.6 kubectl patch deployment nginx --patch-file=patch.yaml kubectl rollout undo deployment/nginx","title":"Exercise 1 - Understand deployments and how to perform rolling update and rollbacks"},{"location":"lab-guide/02-workloads-and-scheduling/#exercise-2-use-configmaps-to-configure-applications","text":"Create a configmap named mycm that has the following key=value pair key = owner value = yourname Create a pod of your choice, such as nginx . Configure this Pod so that the underlying container has the environent varibale OWNER set to the value of this configmap Answer Create configmap: kubectl create configmap mycm --from-literal=owner=david Define Pod: apiVersion: v1 kind: Pod metadata: name: nginx-configmap spec: containers: - name: nginx-configmap image: nginx command: [ \"/bin/sh\", \"-c\", \"env\" ] env: # Define the environment variable - name: OWNER valueFrom: configMapKeyRef: # The ConfigMap containing the value you want to assign to SPECIAL_LEVEL_KEY name: mycm # Specify the key associated with the value key: owner Validate: kubectl logs nginx-configmap | grep OWNER OWNER=david","title":"Exercise 2 - Use ConfigMaps to configure applications"},{"location":"lab-guide/02-workloads-and-scheduling/#exercise-3-use-secrets-to-configure-applications","text":"Create a secret named mysecret that has the following key=value pair dbuser = MyDatabaseUser dbpassword = MyDatabasePassword Create a pod of your choice, such as nginx . Configure this Pod so that the underlying container has the the following environment variables set: DBUSER from secret key dbuser DBPASS from secret key dbpassword Answer kubectl create secret generic mysecret --from-literal=dbuser=\"MyDatabaseUser\" --from-literal=dbpassword=\"MyDatabasePassword\" Apply the following manifest: apiVersion: v1 kind: Pod metadata: name: nginx-secret spec: containers: - name: nginx-secret image: nginx command: [ \"/bin/sh\", \"-c\", \"env\" ] env: - name: dbuser valueFrom: secretKeyRef: name: mysecret key: dbuser - name: dbpassword valueFrom: secretKeyRef: name: mysecret key: dbpassword restartPolicy: Never Which can be validated with: kubectl logs nginx-secret | grep db dbuser=MyDatabaseUser dbpassword=MyDatabasePassword","title":"Exercise 3 - Use Secrets to configure applications"},{"location":"lab-guide/02-workloads-and-scheduling/#exercise-4-know-how-to-scale-applications","text":"Create a deployment object consisting of 3 pods containing a single nginx container Increase this deployment size by adding two additional pods. Decrease this deployment back to the original size of 3 pods Answer - Imperative kubectl create deployment nginx --image=nginx --replicas=3 kubectl scale --replicas=5 deployment nginx kubectl scale --replicas=3 deployment nginx Answer - Declarative Apply initial YAML: apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80 Apply modified YAML: apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: replicas: 5 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1 ports: - containerPort: 80 Apply original YAML: apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80","title":"Exercise 4 - Know how to scale applications"},{"location":"lab-guide/02-workloads-and-scheduling/#excerise-5-understand-how-resource-limits-can-affect-pod-scheduling","text":"Create a new namespace called \"tenant-b-100mi\" Create a memory limit of 100Mi for this namespace Create a pod with a memory request of 150Mi, ensure the limit has been set by verifying you get a error message. Answer kubectl create ns tenant-b-100mi Deploy the manifest: apiVersion: v1 kind: LimitRange metadata: name: tenant-b-memlimit namespace: tenant-b-100mi spec: limits: - max: memory: 100Mi type: Container Test with deploying: apiVersion: v1 kind: Pod metadata: name: default-mem-demo namespace: tenant-b-100mi spec: containers: - name: default-mem-demo image: nginx resources: requests: memory: 150Mi Which should return: The Pod \"default-mem-demo\" is invalid: spec.containers[0].resources.requests: Invalid value: \"150Mi\": must be less than or equal to memory limit","title":"Excerise 5 - Understand how resource limits can affect Pod scheduling"},{"location":"lab-guide/03-services-and-networking/","text":"Lab Exercises for Services & Networking Exercise 0 - Setup Prepare a cluster (Single node, kubeadm, k3s, etc) Open browser tabs to https://kubernetes.io/docs/ , https://github.com/kubernetes/ and https://kubernetes.io/blog/ (these are permitted as per the current guidelines ) Exercise 1 - Understand connectivity between Pods Deploy the following manifest Using kubectl , identify the Pod IP addresses Determine the DNS name of the service. Answer Identify the selector for the service: kubectl describe service nginx-service | grep -i selector Selector: app=nginx Filter kubectl output: kubectl get po -l app=nginx -o wide Service name will be, based on the format [Service Name].[Namespace].[Type].[Base Domain Name] : nginx-service.default.svc.cluster.local Exercise 2 - Understand ClusterIP, NodePort, LoadBalancer service types and endpoints Create three deployments of your choosing Expose one of these deployments with a service of type ClusterIP Expose one of these deployments with a service of type Nodeport Expose one of these deployments with a service of type Loadbalancer Note, this remains in pending status unless your cluster has integration with a cloud provider that provisions one for you (ie AWS ELB), or you have a software implementation such as metallb Answer - Imperative kubectl create deployment nginx-clusterip --image=nginx --replicas 1 kubectl create deployment nginx-nodeport --image=nginx --replicas 1 kubectl create deployment nginx-loadbalancer --image=nginx --replicas 1 kubectl expose deployment nginx-clusterip --type=\"ClusterIP\" --port=\"80\" kubectl expose deployment nginx-nodeport --type=\"NodePort\" --port=\"80\" kubectl expose deployment nginx-loadbalancer --type=\"LoadBalancer\" --port=\"80\" Answer - Declarative Apply the following: kind: Service apiVersion: v1 metadata: name: nginx-clusterip spec: selector: app: nginx-clusterip type: ClusterIP ports: - protocol: TCP port: 80 targetPort: 80 --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-clusterip labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx-clusterip template: metadata: labels: app: nginx-clusterip spec: containers: - name: nginx image: nginx ports: - containerPort: 80 --- kind: Service apiVersion: v1 metadata: name: nginx-nodeport spec: selector: app: nginx-nodeport type: NodePort ports: - protocol: TCP port: 80 targetPort: 80 --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-nodeport labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx-nodeport template: metadata: labels: app: nginx-nodeport spec: containers: - name: nginx image: nginx ports: - containerPort: 80 --- kind: Service apiVersion: v1 metadata: name: nginx-loadbalancer spec: selector: app: nginx-loadbalancer type: LoadBalancer ports: - port: 80 targetPort: 80 --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-loadbalancer labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx-loadbalancer template: metadata: labels: app: nginx-loadbalancer spec: containers: - name: nginx image: nginx ports: - containerPort: 80 Exercise 3 - Know how to use Ingress controllers and Ingress resources Create an ingress object named myingress with the following specification: Manages the host myingress.mydomain Traffic to the base path / will be forwarded to a service called main on port 80 Traffic to the path /api will be forwarded to a service called api on port 8080 Answer - Imperative kubectl create ingress myingress --rule=\"myingress.mydomain/=main:80\" --rule=\"myingress.mydomain/api=api:8080\" Answer - Declarative Apply the following YAML: apiVersion: networking.k8s.io/v1 kind: Ingress metadata: creationTimestamp: null name: myingress spec: rules: - host: myingress.mydomain http: paths: - backend: service: name: main port: number: 80 path: / pathType: Exact - backend: service: name: api port: number: 8080 path: /api pathType: Exact Exercise 4 - Know how to configure and use CoreDNS Identify the configuration location of coredns Modify the coredns config file so DNS queries not resolved by itself are forwarded to the DNS server 8.8.8.8 Validate the changes you have made Add additional configuration so that all DNS queries for custom.local are forwarded to the resolver 10.5.4.223 Answer kubectl get cm coredns -n kube-system NAME DATA AGE coredns 2 94d kubectl edit cm coredns -n kube-system replace: forward . /etc/resolv.conf with forward . 8.8.8.8 Add the block: custom.local:53 { errors cache 30 forward . 10.5.4.223 reload }","title":"Part Three - Services & Networking"},{"location":"lab-guide/03-services-and-networking/#lab-exercises-for-services-networking","text":"","title":"Lab Exercises for Services &amp; Networking"},{"location":"lab-guide/03-services-and-networking/#exercise-0-setup","text":"Prepare a cluster (Single node, kubeadm, k3s, etc) Open browser tabs to https://kubernetes.io/docs/ , https://github.com/kubernetes/ and https://kubernetes.io/blog/ (these are permitted as per the current guidelines )","title":"Exercise 0 - Setup"},{"location":"lab-guide/03-services-and-networking/#exercise-1-understand-connectivity-between-pods","text":"Deploy the following manifest Using kubectl , identify the Pod IP addresses Determine the DNS name of the service. Answer Identify the selector for the service: kubectl describe service nginx-service | grep -i selector Selector: app=nginx Filter kubectl output: kubectl get po -l app=nginx -o wide Service name will be, based on the format [Service Name].[Namespace].[Type].[Base Domain Name] : nginx-service.default.svc.cluster.local","title":"Exercise 1 - Understand connectivity between Pods"},{"location":"lab-guide/03-services-and-networking/#exercise-2-understand-clusterip-nodeport-loadbalancer-service-types-and-endpoints","text":"Create three deployments of your choosing Expose one of these deployments with a service of type ClusterIP Expose one of these deployments with a service of type Nodeport Expose one of these deployments with a service of type Loadbalancer Note, this remains in pending status unless your cluster has integration with a cloud provider that provisions one for you (ie AWS ELB), or you have a software implementation such as metallb Answer - Imperative kubectl create deployment nginx-clusterip --image=nginx --replicas 1 kubectl create deployment nginx-nodeport --image=nginx --replicas 1 kubectl create deployment nginx-loadbalancer --image=nginx --replicas 1 kubectl expose deployment nginx-clusterip --type=\"ClusterIP\" --port=\"80\" kubectl expose deployment nginx-nodeport --type=\"NodePort\" --port=\"80\" kubectl expose deployment nginx-loadbalancer --type=\"LoadBalancer\" --port=\"80\" Answer - Declarative Apply the following: kind: Service apiVersion: v1 metadata: name: nginx-clusterip spec: selector: app: nginx-clusterip type: ClusterIP ports: - protocol: TCP port: 80 targetPort: 80 --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-clusterip labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx-clusterip template: metadata: labels: app: nginx-clusterip spec: containers: - name: nginx image: nginx ports: - containerPort: 80 --- kind: Service apiVersion: v1 metadata: name: nginx-nodeport spec: selector: app: nginx-nodeport type: NodePort ports: - protocol: TCP port: 80 targetPort: 80 --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-nodeport labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx-nodeport template: metadata: labels: app: nginx-nodeport spec: containers: - name: nginx image: nginx ports: - containerPort: 80 --- kind: Service apiVersion: v1 metadata: name: nginx-loadbalancer spec: selector: app: nginx-loadbalancer type: LoadBalancer ports: - port: 80 targetPort: 80 --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-loadbalancer labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx-loadbalancer template: metadata: labels: app: nginx-loadbalancer spec: containers: - name: nginx image: nginx ports: - containerPort: 80","title":"Exercise 2 - Understand ClusterIP, NodePort, LoadBalancer service types and endpoints"},{"location":"lab-guide/03-services-and-networking/#exercise-3-know-how-to-use-ingress-controllers-and-ingress-resources","text":"Create an ingress object named myingress with the following specification: Manages the host myingress.mydomain Traffic to the base path / will be forwarded to a service called main on port 80 Traffic to the path /api will be forwarded to a service called api on port 8080 Answer - Imperative kubectl create ingress myingress --rule=\"myingress.mydomain/=main:80\" --rule=\"myingress.mydomain/api=api:8080\" Answer - Declarative Apply the following YAML: apiVersion: networking.k8s.io/v1 kind: Ingress metadata: creationTimestamp: null name: myingress spec: rules: - host: myingress.mydomain http: paths: - backend: service: name: main port: number: 80 path: / pathType: Exact - backend: service: name: api port: number: 8080 path: /api pathType: Exact","title":"Exercise 3 - Know how to use Ingress controllers and Ingress resources"},{"location":"lab-guide/03-services-and-networking/#exercise-4-know-how-to-configure-and-use-coredns","text":"Identify the configuration location of coredns Modify the coredns config file so DNS queries not resolved by itself are forwarded to the DNS server 8.8.8.8 Validate the changes you have made Add additional configuration so that all DNS queries for custom.local are forwarded to the resolver 10.5.4.223 Answer kubectl get cm coredns -n kube-system NAME DATA AGE coredns 2 94d kubectl edit cm coredns -n kube-system replace: forward . /etc/resolv.conf with forward . 8.8.8.8 Add the block: custom.local:53 { errors cache 30 forward . 10.5.4.223 reload }","title":"Exercise 4 - Know how to configure and use CoreDNS"},{"location":"lab-guide/04-storage/","text":"Lab Exercises for Storage Exercise 0 - Setup Prepare a cluster (Single node, kubeadm, k3s, etc) Open browser tabs to Kubernetes Documentation , Kubernetes GitHub and Kubernetes Blog (these are permitted as per the current guidelines ) Exercise 1 - Understand storage classes, persistent volumes Note: Leverage the docs to help you. It's unlikely you will need to recall the nuances from memory - Kubernetes Storage Classes Create a storageclass object named aws-ebs with the following specification: The provisioner should be kubernetes.io/aws-ebs The type is io1 The reclaimPolicy is Delete allowVolumeExpansion set to false Create a persistentvolume based on this storage class Note, it will not initialize unless AWS integration is configured, but the objective is to get the YAML correct. Answer Apply the following YAML to create the Storage Class apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: aws-ebs provisioner: kubernetes.io/aws-ebs parameters: type: io1 reclaimPolicy: Delete allowVolumeExpansion: false mountOptions: - debug volumeBindingMode: Immediate Apply the following YAML to create a Persistent Volume based on the Storage Class apiVersion: v1 kind: PersistentVolume metadata: name: pv-test spec: capacity: storage: 5Gi volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Recycle storageClassName: aws-ebs awsElasticBlockStore: fsType: \"ext4\" volumeID: \"vol-id\" validate with: kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pv-test 5Gi RWO Recycle Available aws-ebs 5 Exercise 2 - Know how to configure applications with persistent storage Create a persistentVolume object of type hostPath with the following parameters: 1GB Capacity Path on the host is /tmp storageClassName is Manual accessModes is ReadWriteOnce Create a persistentVolumeClaim to the aforementioned persistentVolume Create a pod workload to leverage this `persistentVolumeClaim To create the StorageClass manual apply the following: apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: manual provisioner: kubernetes.io/no-provisioner Answer apiVersion: v1 kind: PersistentVolume metadata: name: pv-hostpath-1gb spec: capacity: storage: 1Gi volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Recycle storageClassName: manual hostPath: path: /tmp --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-hostpath-claim spec: accessModes: - ReadWriteOnce volumeMode: Filesystem resources: requests: storage: 512Mi storageClassName: manual --- apiVersion: v1 kind: Pod metadata: name: pod-with-pvc spec: volumes: - name: myvol persistentVolumeClaim: claimName: pvc-hostpath-claim containers: - name: busybox image: busybox args: - sleep - \"1000000\" volumeMounts: - mountPath: \"/mnt/readonly\" name: myvol Validate with: kubectl get po,pv,pvc NAME READY STATUS RESTARTS AGE pod/pod-with-pvc 1/1 Running 0 2m7s NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pv-hostpath-1gb 1Gi RWO Recycle Bound default/pvc-hostpath-claim manual 2m7s NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/pvc-hostpath-claim Bound pv-hostpath-1gb 1Gi RWO manual 2m7s","title":"Part Four - Storage"},{"location":"lab-guide/04-storage/#lab-exercises-for-storage","text":"","title":"Lab Exercises for Storage"},{"location":"lab-guide/04-storage/#exercise-0-setup","text":"Prepare a cluster (Single node, kubeadm, k3s, etc) Open browser tabs to Kubernetes Documentation , Kubernetes GitHub and Kubernetes Blog (these are permitted as per the current guidelines )","title":"Exercise 0 - Setup"},{"location":"lab-guide/04-storage/#exercise-1-understand-storage-classes-persistent-volumes","text":"Note: Leverage the docs to help you. It's unlikely you will need to recall the nuances from memory - Kubernetes Storage Classes Create a storageclass object named aws-ebs with the following specification: The provisioner should be kubernetes.io/aws-ebs The type is io1 The reclaimPolicy is Delete allowVolumeExpansion set to false Create a persistentvolume based on this storage class Note, it will not initialize unless AWS integration is configured, but the objective is to get the YAML correct. Answer Apply the following YAML to create the Storage Class apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: aws-ebs provisioner: kubernetes.io/aws-ebs parameters: type: io1 reclaimPolicy: Delete allowVolumeExpansion: false mountOptions: - debug volumeBindingMode: Immediate Apply the following YAML to create a Persistent Volume based on the Storage Class apiVersion: v1 kind: PersistentVolume metadata: name: pv-test spec: capacity: storage: 5Gi volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Recycle storageClassName: aws-ebs awsElasticBlockStore: fsType: \"ext4\" volumeID: \"vol-id\" validate with: kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pv-test 5Gi RWO Recycle Available aws-ebs 5","title":"Exercise 1 - Understand storage classes, persistent volumes"},{"location":"lab-guide/04-storage/#exercise-2-know-how-to-configure-applications-with-persistent-storage","text":"Create a persistentVolume object of type hostPath with the following parameters: 1GB Capacity Path on the host is /tmp storageClassName is Manual accessModes is ReadWriteOnce Create a persistentVolumeClaim to the aforementioned persistentVolume Create a pod workload to leverage this `persistentVolumeClaim To create the StorageClass manual apply the following: apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: manual provisioner: kubernetes.io/no-provisioner Answer apiVersion: v1 kind: PersistentVolume metadata: name: pv-hostpath-1gb spec: capacity: storage: 1Gi volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Recycle storageClassName: manual hostPath: path: /tmp --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-hostpath-claim spec: accessModes: - ReadWriteOnce volumeMode: Filesystem resources: requests: storage: 512Mi storageClassName: manual --- apiVersion: v1 kind: Pod metadata: name: pod-with-pvc spec: volumes: - name: myvol persistentVolumeClaim: claimName: pvc-hostpath-claim containers: - name: busybox image: busybox args: - sleep - \"1000000\" volumeMounts: - mountPath: \"/mnt/readonly\" name: myvol Validate with: kubectl get po,pv,pvc NAME READY STATUS RESTARTS AGE pod/pod-with-pvc 1/1 Running 0 2m7s NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pv-hostpath-1gb 1Gi RWO Recycle Bound default/pvc-hostpath-claim manual 2m7s NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/pvc-hostpath-claim Bound pv-hostpath-1gb 1Gi RWO manual 2m7s","title":"Exercise 2 - Know how to configure applications with persistent storage"},{"location":"lab-guide/05-troubleshooting/","text":"Lab Exercises for Troubleshooting Exercise 0 - Setup Prepare a cluster (Single node, kubeadm, k3s, etc) Open browser tabs to Kubernetes Documentation , Kubernetes GitHub and Kubernetes Blog (these are permitted as per the current guidelines ) Ensure your etcd nodes have etcdctl installed Exercise 1 - Evaluate cluster and node logging For your cluster type, determine how to acquire logs for your master nodes. They could be in the form of: Services Static Pods Kubernetes Pods Using kubectl get a list of events from your cluster Using etcdctl determine the health of the etcd cluster Answer 1 Is dependent on how the cluster was made and potentially which OS's were used. For example, if K8S components manifest as Kubernetes Pods: kubectl logs <podname> <namespace> kubectl get events etcdctl --write-out=table --endpoints=$ENDPOINTS endpoint status +------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | https://127.0.0.1:2379 | 4e30a295f2c3c1a4 | 3.5.0 | 8.1 MB | true | false | 3 | 7903 | 7903 | | +------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ Exercise 2 - Understand how to monitor applications Deploy the following manifest: counter-pod.yaml Acquire the logs from this pod - what do they mention? Answer kubectl logs counter 0: Mon Dec 9 13:15:08 UTC 2024 1: Mon Dec 9 13:15:09 UTC 2024 2: Mon Dec 9 13:15:10 UTC 2024 3: Mon Dec 9 13:15:11 UTC 2024 4: Mon Dec 9 13:15:12 UTC 2024 5: Mon Dec 9 13:15:13 UTC 2024 6: Mon Dec 9 13:15:14 UTC 2024 7: Mon Dec 9 13:15:15 UTC 2024 8: Mon Dec 9 13:15:16 UTC 2024 9: Mon Dec 9 13:15:17 UTC 2024 Exercise 3 - Troubleshoot application failure Deploy the following manifest: brokenpod.yaml . It will create a Pod named nginx-pod . Determine why this Pod will not enter running state by using kubectl Answer kubectl describe pod nginx-pod .. Normal BackOff 4m55s (x7 over 6m56s) kubelet Back-off pulling image \"nginx:invalidversion\" .. Exercise 4 - Troubleshoot networking Deploy the following manifest: https://raw.githubusercontent.com/David-VTUK/CKAExampleYaml/master/nginx-svc-and-deployment-broken.yaml , It will create deployment and service objects. Identify the DNS name of this service. Test resolution of this DNS record: Create a Pod that has nslookup installed. ie: kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml Test sending traffic to this service Create a Pod that has curl installed. ie: kubectl run curl --image=radial/busyboxplus:curl -i --tty - Why does it fail? Rectify the identified error in step 3 Answer nginx-service.default.svc.cluster.local kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml kubectl exec -it dnsutils sh / # nslookup nginx-service.default.svc.cluster.local Server: 10.96.0.10 Address: 10.96.0.10#53 Name: nginx-service.default.svc.cluster.local Address: 10.99.41.254 kubectl run curl --image=radial/busyboxplus:curl -i --tty If you don't see a command prompt, try pressing enter. [ root@curl:/ ]$ curl nginx-service.default.svc.cluster.local curl: (7) Failed to connect to nginx-service.default.svc.cluster.local port 80: Connection refused Check service: kubectl describe service nginx-service Name: nginx-service Namespace: default Labels: <none> Annotations: <none> Selector: app=nginx Type: ClusterIP IP Families: <none> IP: 10.99.41.254 IPs: 10.99.41.254 Port: <unset> 80/TCP TargetPort: 8080/TCP Endpoints: 10.244.1.18:8080,10.244.1.19:8080,10.244.1.20:8080 Session Affinity: None Events: <none> Note: Service is listening on port 80 Service has a endpoint list, with target port of 8080 Test curl directly against pod: curl 10.244.1.18:8080 curl: (7) Failed to connect to 10.244.1.18 port 8080: Connection refused Port 8080 isn't listening, check the pod config: kubectl describe po nginx-deployment-5d59d67564-bk9xb | grep -i \"port:\" Port: 80/TCP The service is trying to forward traffic to port 8080 on the container, but the container is only listening on port 80. Reconfigure the service object, ie: kubectl edit service nginx-service Replace targetPort: 8080 with targetPort: 80 Retest: curl nginx-service.default.svc.cluster.local <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title>","title":"Part Five - Troubleshooting"},{"location":"lab-guide/05-troubleshooting/#lab-exercises-for-troubleshooting","text":"","title":"Lab Exercises for Troubleshooting"},{"location":"lab-guide/05-troubleshooting/#exercise-0-setup","text":"Prepare a cluster (Single node, kubeadm, k3s, etc) Open browser tabs to Kubernetes Documentation , Kubernetes GitHub and Kubernetes Blog (these are permitted as per the current guidelines ) Ensure your etcd nodes have etcdctl installed","title":"Exercise 0 - Setup"},{"location":"lab-guide/05-troubleshooting/#exercise-1-evaluate-cluster-and-node-logging","text":"For your cluster type, determine how to acquire logs for your master nodes. They could be in the form of: Services Static Pods Kubernetes Pods Using kubectl get a list of events from your cluster Using etcdctl determine the health of the etcd cluster Answer 1 Is dependent on how the cluster was made and potentially which OS's were used. For example, if K8S components manifest as Kubernetes Pods: kubectl logs <podname> <namespace> kubectl get events etcdctl --write-out=table --endpoints=$ENDPOINTS endpoint status +------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | https://127.0.0.1:2379 | 4e30a295f2c3c1a4 | 3.5.0 | 8.1 MB | true | false | 3 | 7903 | 7903 | | +------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+","title":"Exercise 1 - Evaluate cluster and node logging"},{"location":"lab-guide/05-troubleshooting/#exercise-2-understand-how-to-monitor-applications","text":"Deploy the following manifest: counter-pod.yaml Acquire the logs from this pod - what do they mention? Answer kubectl logs counter 0: Mon Dec 9 13:15:08 UTC 2024 1: Mon Dec 9 13:15:09 UTC 2024 2: Mon Dec 9 13:15:10 UTC 2024 3: Mon Dec 9 13:15:11 UTC 2024 4: Mon Dec 9 13:15:12 UTC 2024 5: Mon Dec 9 13:15:13 UTC 2024 6: Mon Dec 9 13:15:14 UTC 2024 7: Mon Dec 9 13:15:15 UTC 2024 8: Mon Dec 9 13:15:16 UTC 2024 9: Mon Dec 9 13:15:17 UTC 2024","title":"Exercise 2 - Understand how to monitor applications"},{"location":"lab-guide/05-troubleshooting/#exercise-3-troubleshoot-application-failure","text":"Deploy the following manifest: brokenpod.yaml . It will create a Pod named nginx-pod . Determine why this Pod will not enter running state by using kubectl Answer kubectl describe pod nginx-pod .. Normal BackOff 4m55s (x7 over 6m56s) kubelet Back-off pulling image \"nginx:invalidversion\" ..","title":"Exercise 3 - Troubleshoot application failure"},{"location":"lab-guide/05-troubleshooting/#exercise-4-troubleshoot-networking","text":"Deploy the following manifest: https://raw.githubusercontent.com/David-VTUK/CKAExampleYaml/master/nginx-svc-and-deployment-broken.yaml , It will create deployment and service objects. Identify the DNS name of this service. Test resolution of this DNS record: Create a Pod that has nslookup installed. ie: kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml Test sending traffic to this service Create a Pod that has curl installed. ie: kubectl run curl --image=radial/busyboxplus:curl -i --tty - Why does it fail? Rectify the identified error in step 3 Answer nginx-service.default.svc.cluster.local kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml kubectl exec -it dnsutils sh / # nslookup nginx-service.default.svc.cluster.local Server: 10.96.0.10 Address: 10.96.0.10#53 Name: nginx-service.default.svc.cluster.local Address: 10.99.41.254 kubectl run curl --image=radial/busyboxplus:curl -i --tty If you don't see a command prompt, try pressing enter. [ root@curl:/ ]$ curl nginx-service.default.svc.cluster.local curl: (7) Failed to connect to nginx-service.default.svc.cluster.local port 80: Connection refused Check service: kubectl describe service nginx-service Name: nginx-service Namespace: default Labels: <none> Annotations: <none> Selector: app=nginx Type: ClusterIP IP Families: <none> IP: 10.99.41.254 IPs: 10.99.41.254 Port: <unset> 80/TCP TargetPort: 8080/TCP Endpoints: 10.244.1.18:8080,10.244.1.19:8080,10.244.1.20:8080 Session Affinity: None Events: <none> Note: Service is listening on port 80 Service has a endpoint list, with target port of 8080 Test curl directly against pod: curl 10.244.1.18:8080 curl: (7) Failed to connect to 10.244.1.18 port 8080: Connection refused Port 8080 isn't listening, check the pod config: kubectl describe po nginx-deployment-5d59d67564-bk9xb | grep -i \"port:\" Port: 80/TCP The service is trying to forward traffic to port 8080 on the container, but the container is only listening on port 80. Reconfigure the service object, ie: kubectl edit service nginx-service Replace targetPort: 8080 with targetPort: 80 Retest: curl nginx-service.default.svc.cluster.local <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title>","title":"Exercise 4 - Troubleshoot networking"},{"location":"revision-topics/01-architcture-installation-configuration/","text":"Cluster Architecture, Installation & Configuration Manage Role Based Access Control (RBAC) Kubernetes implements an RBAC framework to govern access to resources within a cluster and forms part of the overall Authentication, Authorization and Admission control framework. To determine who (or what) has access to which resources, a number of steps have to be executed. Step 1 - Authentication First step is Authentication which is how a user or service account identifies itself. Depending on the source, a corresponding authentication module is used. Authentication modules include the ability to authenticate from the following: Client Certificate Password Plain Tokens Bootstrap Tokens JWT Tokens (for service accounts) All authentication is handled via HTTP over TLS. Step 2 - Authorization After a user or service account is authenticated, the request must then be authorized. Any authentication request is followed by some kind of action request, and the action defines the object(s) that request needs to apply to, and what the action is. For example, to list the pods in a given namespace. Any and all requests are facilitated providing an existing policy gives the user those permissions. Steps 3 & 4 - Admission Control Admission Control Modules are software modules that can modify or reject requests. In addition to the attributes available to Authorization Modules, Admission Control Modules can access the contents of the object that is being created or updated. They act on objects being created, deleted, updated or connected (proxy), but not reads. Role and Rolebindings Implementing RBAC rules largely involves two object types within Kubernetes - role and rolebindings : A role grants access to resources within a single namespace. A rolebinding grants the permissions from a role to a user, group or service account within a single namespace. clusterrole and clusterrolebindings operate similarly, but obviously provide access to non-namespaced resources. kubectl api-resources --namespaced=false can be used to determine which resource types are not namespaced. Examples include: node , persistentvolume , storageclass and users . Users can either be serviceaccounts or users . The former is typically used to authenticate applications, the latter for human users. To test, the below creates namespace , serviceaccount , role and rolebinding apiVersion: v1 kind: Namespace metadata: name: rbac-test apiVersion: v1 kind: ServiceAccount metadata: name: rbac-test-sa namespace: rbac-test ``` of particular importance is the format of the below. `apiGroup` : Determines which API group to apply this to. `resources`: Which resource types to apply this to. `verbs`: What we can do to these objects (ie create, delete, watch, etc) ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: rbac-test-role namespace: rbac-test rules: - apiGroups: [\"\"] resources: [\"pods\"] verbs: [\"get\", \"list\", \"watch\"] apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: rbac-test-rolebinding namespace: rbac-test roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: rbac-test-role subjects: - kind: ServiceAccount name: rbac-test-sa namespace: rbac-test We can then validate this with kubectl. The following returns yes as that service account can get pods kubectl -n rbac-test --as=system:serviceaccount:rbac-test:rbac-test-sa auth can-i get pods yes However with secrets , it returns no kubectl -n rbac-test --as=system:serviceaccount:rbac-test:rbac-test-sa auth can-i get secrets no Use Kubeadm to install a basic cluster kubeadm is a utility to bootstrap Kubernetes to a number of existing, vanilla nodes. It takes care of the etcd cluster, Kubernetes master and worker nodes including all the required components to instantiate a viable minimum k8s cluster. What you get at the end of using kubeadm is a fully working, fully functioning kubernetes cluster. It's at the opposite end of the spectrum in terms of difficulty compared to, for example, Kelsey Hightower's \"Kubernetes the hard way\". For the exam, it is recommended that you become familiar with both ways of deploying Kubernetes clusters. Kubeadm is a command line utility that performs the following functions: kubeadm init to bootstrap a Kubernetes control-plane node kubeadm join to bootstrap a Kubernetes worker node and join it to the cluster kubeadm upgrade to upgrade a Kubernetes cluster to a newer version kubeadm config if you initialized your cluster using kubeadm v1.7.x or lower, to configure your cluster for kubeadm upgrade kubeadm token to manage tokens for kubeadm join kubeadm reset to revert any changes made to this host by kubeadm init or kubeadm join kubeadm version to print the kubeadm version kubeadm alpha to preview a set of features made available for gathering feedback from the community Kubeadm - Master Node Install In the following examples 3x Ubuntu Server VMs were created k8s-cl02-ms01 k8s-cl02-wk01 k8s-cl02-wk02 Where appropriate, ensure your nodes have a container runtime installed. On the master node install the required binaries apt-get update && apt-get install -y apt-transport-https curl curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - cat <<EOF >/etc/apt/sources.list.d/kubernetes.list deb https://apt.kubernetes.io/ kubernetes-xenial main EOF apt-get update apt-get install -y kubelet kubeadm kubectl apt-mark hold kubelet kubeadm kubectl Initialise the master node: sudo kubeadm init --pod-network-cidr=10.244.0.0/16 Note, the requirement to pass --pod-network is dependent on the chosen CNI. For Flannel, this is required. Kubeadm will also let you know if any prerequisites are not made. Once completed, a message will be displayed: Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run \"kubectl apply -f [podnetwork].yaml\" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 172.16.10.80:6443 --token j5nqhd.cnfmnjgc68aato60 \\ --discovery-token-ca-cert-hash sha256:cbc91031c1ffa47bbea83aa1cf65e99821a1f582c4363e1a4408715bfd66bb60 Some important pieces of information to note: Kubeadm has created the admin kubeconfig file for you, and recommends copying this to the logged on users home directory for ease Kubeadm has not deployed a pod networking solution yet. Therefore, this is a post-install activity Kubeadm has provided a join command together with a token to add worker nodes. We can regenerate this token if required. If we issue a kubectl get nodes command we will see the master node is not ready NAME STATUS ROLES AGE VERSION k8s-cl02-ms01 NotReady master 6m20s v1.20.2 As per the output of kubeadm, install a network solution, Ie flannel. For flannel to work correctly, you must pass --pod-network-cidr=10.244.0.0/16 to kubeadm init. Additionally, set /proc/sys/net/bridge/bridge-nf-call-iptables to 1 by running sysctl net.bridge.bridge-nf-call-iptables=1 . Install Flannel: kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml After a few seconds, the master node will now be ready NAME STATUS ROLES AGE VERSION k8s-cl02-ms01 Ready master 10m v1.20.2 Kubeadm - Install worker nodes The installation process for worker nodes is similar to master nodes - the only exception is we do not execute the \"kubeadm init\" command, as this is only run on masters. For workers, we use \"kubeadm join\". As prep: Install a container runtime Install the kubeadm binaries (as above) To join a worker node to a cluster created by kubeadm we need to use the kubeadm join command with a token generated on the master. This is shown after we run kubeadm init on the master node. However, we can easily regenerate this on the master node should it not be noted down or expired: (on the master node) david@k8s-cl02-ms01:~$ kubeadm token create --print-join-command kubeadm join 172.16.10.80:6443 --token ht55yv.8lq69q0189xhe2ql --discovery-token-ca-cert-hash sha256:cbc91031c1ffa47bbea83aa1cf65e99821a1f582c4363e1a4408715bfd66bb60 Use this command on the worker (as root) root@k8s-cl02-wk01:~# kubeadm join 172.16.10.80:6443 --token ht55yv.8lq69q0189xhe2ql --discovery-token-ca-cert-hash sha256:cbc91031c1ffa47bbea83aa1cf65e99821a1f582c4363e1a4408715bfd66bb60 After which confirmation will be displayed: This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. To validate, run kubectl get nodes on the master node: NAME STATUS ROLES AGE VERSION k8s-cl02-ms01 Ready master 50m v1.20.2 k8s-cl02-wk01 Ready <none> 2m10s v1.20.2 Manage a highly-available Kubernetes cluster The previous section demonstrated creating a K8s cluster with one master node and several worker nodes - this does not provide resilience for the control plane. Several topologies exist for doing so: Stacked etcd Multiple worker nodes Multiple control plane nodes fronted by a loadbalancer Embedded etcd within control plane Notes: etcd is quorum based. Therefore, if using stacked control plane nodes with etcd, odd numbers must be used. External etcd Notes: Multiple worker nodes Multiple control plane nodes fronted by a loadbalancer Etcd is external of the k8s cluster Notes: Advantage with this setup is etcd and the control plane can be scaled and managed independently of each other. This provides greater flexibility at the expense of operational complexity. Assessing cluster health kubectl get componentstatus is deprecated as of 1.20. A suitable replacement includes probing the API server directly, For example, on a master node, run curl -k https://localhost:6443/livez?verbose which returns: [+]ping ok [+]log ok [+]etcd ok [+]poststarthook/start-kube-apiserver-admission-initializer ok [+]poststarthook/generic-apiserver-start-informers ok .....etc Three endpoints exist - healthz , livez and readyz to indicate the current status of the API server Provision underlying infrastructure to deploy a Kubernetes cluster The topology choices above will influence the underlying resources that need to be provisioned. How these are provisioned are specific to the underlying cloud provider. Some generic observations: Disable swap. Leverage cloud capabilities for HA - ie using multiple AZ's. Windows can be used for worker nodes, but not control plane. Perform a version upgrade on a Kubernetes cluster using Kubeadm First, install kubeadm to a specific version. This will determine the k8s version that it deploys: sudo apt-get update && sudo apt-get install -y kubeadm=1.19.0-00 kubelet=1.19.0-00 kubectl=1.19.0-00 && sudo apt-mark hold kubeadm Stand up a k8s cluster sudo kubeadm init --pod-network-cidr=10.244.0.0/16 Add CNI https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml To upgrade the underlying k8s cluster, we need to upgrade kubeadm. Update kubeadm sudo apt-mark unhold kubeadm sudo apt-get install --only-upgrade kubeadm Next we plan the upgrade - this won't change our cluster but will display what changes can be made: sudo kubeadm upgrade plan Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': COMPONENT CURRENT AVAILABLE kubelet 1 x v1.19.0 v1.20.2 Upgrade to the latest stable version: COMPONENT CURRENT AVAILABLE kube-apiserver v1.19.7 v1.20.2 kube-controller-manager v1.19.7 v1.20.2 kube-scheduler v1.19.7 v1.20.2 kube-proxy v1.19.7 v1.20.2 CoreDNS 1.7.0 1.7.0 etcd 3.4.9-1 3.4.13-0 You can now apply the upgrade by executing the following command: kubeadm upgrade apply v1.20.2 Important note: kubelet must be upgraded manually after this step. Upgrade the cluster: kubeadm upgrade apply v1.20.2 upgrade Kubelet: sudo apt-get install --only-upgrade kubelet kubectl Implement etcd backup and restore Backing up etcd Take a snapshot of the DB, then store it in a safe location: ETCDCTL_API=3 etcdctl snapshot save snapshot.db --cacert /etc/kubernetes/pki/etcd/server.crt --cert /etc/kubernetes/pki/etcd/ca.crt --key /etc/kubernetes/pki/etcd/ca.key Verify the backup: sudo ETCDCTL_API=3 etcdctl --write-out=table snapshot status snapshot.db +----------+----------+------------+------------+ | HASH | REVISION | TOTAL KEYS | TOTAL SIZE | +----------+----------+------------+------------+ | 2125d542 | 364069 | 770 | 3.8 MB | +----------+----------+------------+------------+ Restore to etcd To perform a restore: ETCDCTL_API=3 etcdctl snapshot restore snapshot.db","title":"Part One - Architecture, Installation and Configuration"},{"location":"revision-topics/01-architcture-installation-configuration/#cluster-architecture-installation-configuration","text":"","title":"Cluster Architecture, Installation &amp; Configuration"},{"location":"revision-topics/01-architcture-installation-configuration/#manage-role-based-access-control-rbac","text":"Kubernetes implements an RBAC framework to govern access to resources within a cluster and forms part of the overall Authentication, Authorization and Admission control framework. To determine who (or what) has access to which resources, a number of steps have to be executed.","title":"Manage Role Based Access Control (RBAC)"},{"location":"revision-topics/01-architcture-installation-configuration/#step-1-authentication","text":"First step is Authentication which is how a user or service account identifies itself. Depending on the source, a corresponding authentication module is used. Authentication modules include the ability to authenticate from the following: Client Certificate Password Plain Tokens Bootstrap Tokens JWT Tokens (for service accounts) All authentication is handled via HTTP over TLS.","title":"Step 1 - Authentication"},{"location":"revision-topics/01-architcture-installation-configuration/#step-2-authorization","text":"After a user or service account is authenticated, the request must then be authorized. Any authentication request is followed by some kind of action request, and the action defines the object(s) that request needs to apply to, and what the action is. For example, to list the pods in a given namespace. Any and all requests are facilitated providing an existing policy gives the user those permissions.","title":"Step 2 - Authorization"},{"location":"revision-topics/01-architcture-installation-configuration/#steps-3-4-admission-control","text":"Admission Control Modules are software modules that can modify or reject requests. In addition to the attributes available to Authorization Modules, Admission Control Modules can access the contents of the object that is being created or updated. They act on objects being created, deleted, updated or connected (proxy), but not reads.","title":"Steps 3 &amp; 4 - Admission Control"},{"location":"revision-topics/01-architcture-installation-configuration/#role-and-rolebindings","text":"Implementing RBAC rules largely involves two object types within Kubernetes - role and rolebindings : A role grants access to resources within a single namespace. A rolebinding grants the permissions from a role to a user, group or service account within a single namespace. clusterrole and clusterrolebindings operate similarly, but obviously provide access to non-namespaced resources. kubectl api-resources --namespaced=false can be used to determine which resource types are not namespaced. Examples include: node , persistentvolume , storageclass and users . Users can either be serviceaccounts or users . The former is typically used to authenticate applications, the latter for human users. To test, the below creates namespace , serviceaccount , role and rolebinding apiVersion: v1 kind: Namespace metadata: name: rbac-test apiVersion: v1 kind: ServiceAccount metadata: name: rbac-test-sa namespace: rbac-test ``` of particular importance is the format of the below. `apiGroup` : Determines which API group to apply this to. `resources`: Which resource types to apply this to. `verbs`: What we can do to these objects (ie create, delete, watch, etc) ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: rbac-test-role namespace: rbac-test rules: - apiGroups: [\"\"] resources: [\"pods\"] verbs: [\"get\", \"list\", \"watch\"] apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: rbac-test-rolebinding namespace: rbac-test roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: rbac-test-role subjects: - kind: ServiceAccount name: rbac-test-sa namespace: rbac-test We can then validate this with kubectl. The following returns yes as that service account can get pods kubectl -n rbac-test --as=system:serviceaccount:rbac-test:rbac-test-sa auth can-i get pods yes However with secrets , it returns no kubectl -n rbac-test --as=system:serviceaccount:rbac-test:rbac-test-sa auth can-i get secrets no","title":"Role and Rolebindings"},{"location":"revision-topics/01-architcture-installation-configuration/#use-kubeadm-to-install-a-basic-cluster","text":"kubeadm is a utility to bootstrap Kubernetes to a number of existing, vanilla nodes. It takes care of the etcd cluster, Kubernetes master and worker nodes including all the required components to instantiate a viable minimum k8s cluster. What you get at the end of using kubeadm is a fully working, fully functioning kubernetes cluster. It's at the opposite end of the spectrum in terms of difficulty compared to, for example, Kelsey Hightower's \"Kubernetes the hard way\". For the exam, it is recommended that you become familiar with both ways of deploying Kubernetes clusters. Kubeadm is a command line utility that performs the following functions: kubeadm init to bootstrap a Kubernetes control-plane node kubeadm join to bootstrap a Kubernetes worker node and join it to the cluster kubeadm upgrade to upgrade a Kubernetes cluster to a newer version kubeadm config if you initialized your cluster using kubeadm v1.7.x or lower, to configure your cluster for kubeadm upgrade kubeadm token to manage tokens for kubeadm join kubeadm reset to revert any changes made to this host by kubeadm init or kubeadm join kubeadm version to print the kubeadm version kubeadm alpha to preview a set of features made available for gathering feedback from the community","title":"Use Kubeadm to install a basic cluster"},{"location":"revision-topics/01-architcture-installation-configuration/#kubeadm-master-node-install","text":"In the following examples 3x Ubuntu Server VMs were created k8s-cl02-ms01 k8s-cl02-wk01 k8s-cl02-wk02 Where appropriate, ensure your nodes have a container runtime installed. On the master node install the required binaries apt-get update && apt-get install -y apt-transport-https curl curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - cat <<EOF >/etc/apt/sources.list.d/kubernetes.list deb https://apt.kubernetes.io/ kubernetes-xenial main EOF apt-get update apt-get install -y kubelet kubeadm kubectl apt-mark hold kubelet kubeadm kubectl Initialise the master node: sudo kubeadm init --pod-network-cidr=10.244.0.0/16 Note, the requirement to pass --pod-network is dependent on the chosen CNI. For Flannel, this is required. Kubeadm will also let you know if any prerequisites are not made. Once completed, a message will be displayed: Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run \"kubectl apply -f [podnetwork].yaml\" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 172.16.10.80:6443 --token j5nqhd.cnfmnjgc68aato60 \\ --discovery-token-ca-cert-hash sha256:cbc91031c1ffa47bbea83aa1cf65e99821a1f582c4363e1a4408715bfd66bb60 Some important pieces of information to note: Kubeadm has created the admin kubeconfig file for you, and recommends copying this to the logged on users home directory for ease Kubeadm has not deployed a pod networking solution yet. Therefore, this is a post-install activity Kubeadm has provided a join command together with a token to add worker nodes. We can regenerate this token if required. If we issue a kubectl get nodes command we will see the master node is not ready NAME STATUS ROLES AGE VERSION k8s-cl02-ms01 NotReady master 6m20s v1.20.2 As per the output of kubeadm, install a network solution, Ie flannel. For flannel to work correctly, you must pass --pod-network-cidr=10.244.0.0/16 to kubeadm init. Additionally, set /proc/sys/net/bridge/bridge-nf-call-iptables to 1 by running sysctl net.bridge.bridge-nf-call-iptables=1 . Install Flannel: kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml After a few seconds, the master node will now be ready NAME STATUS ROLES AGE VERSION k8s-cl02-ms01 Ready master 10m v1.20.2","title":"Kubeadm - Master Node Install"},{"location":"revision-topics/01-architcture-installation-configuration/#kubeadm-install-worker-nodes","text":"The installation process for worker nodes is similar to master nodes - the only exception is we do not execute the \"kubeadm init\" command, as this is only run on masters. For workers, we use \"kubeadm join\". As prep: Install a container runtime Install the kubeadm binaries (as above) To join a worker node to a cluster created by kubeadm we need to use the kubeadm join command with a token generated on the master. This is shown after we run kubeadm init on the master node. However, we can easily regenerate this on the master node should it not be noted down or expired: (on the master node) david@k8s-cl02-ms01:~$ kubeadm token create --print-join-command kubeadm join 172.16.10.80:6443 --token ht55yv.8lq69q0189xhe2ql --discovery-token-ca-cert-hash sha256:cbc91031c1ffa47bbea83aa1cf65e99821a1f582c4363e1a4408715bfd66bb60 Use this command on the worker (as root) root@k8s-cl02-wk01:~# kubeadm join 172.16.10.80:6443 --token ht55yv.8lq69q0189xhe2ql --discovery-token-ca-cert-hash sha256:cbc91031c1ffa47bbea83aa1cf65e99821a1f582c4363e1a4408715bfd66bb60 After which confirmation will be displayed: This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. To validate, run kubectl get nodes on the master node: NAME STATUS ROLES AGE VERSION k8s-cl02-ms01 Ready master 50m v1.20.2 k8s-cl02-wk01 Ready <none> 2m10s v1.20.2","title":"Kubeadm - Install worker nodes"},{"location":"revision-topics/01-architcture-installation-configuration/#manage-a-highly-available-kubernetes-cluster","text":"The previous section demonstrated creating a K8s cluster with one master node and several worker nodes - this does not provide resilience for the control plane. Several topologies exist for doing so:","title":"Manage a highly-available Kubernetes cluster"},{"location":"revision-topics/01-architcture-installation-configuration/#stacked-etcd","text":"Multiple worker nodes Multiple control plane nodes fronted by a loadbalancer Embedded etcd within control plane Notes: etcd is quorum based. Therefore, if using stacked control plane nodes with etcd, odd numbers must be used.","title":"Stacked etcd"},{"location":"revision-topics/01-architcture-installation-configuration/#external-etcd","text":"Notes: Multiple worker nodes Multiple control plane nodes fronted by a loadbalancer Etcd is external of the k8s cluster Notes: Advantage with this setup is etcd and the control plane can be scaled and managed independently of each other. This provides greater flexibility at the expense of operational complexity.","title":"External etcd"},{"location":"revision-topics/01-architcture-installation-configuration/#assessing-cluster-health","text":"kubectl get componentstatus is deprecated as of 1.20. A suitable replacement includes probing the API server directly, For example, on a master node, run curl -k https://localhost:6443/livez?verbose which returns: [+]ping ok [+]log ok [+]etcd ok [+]poststarthook/start-kube-apiserver-admission-initializer ok [+]poststarthook/generic-apiserver-start-informers ok .....etc Three endpoints exist - healthz , livez and readyz to indicate the current status of the API server","title":"Assessing cluster health"},{"location":"revision-topics/01-architcture-installation-configuration/#provision-underlying-infrastructure-to-deploy-a-kubernetes-cluster","text":"The topology choices above will influence the underlying resources that need to be provisioned. How these are provisioned are specific to the underlying cloud provider. Some generic observations: Disable swap. Leverage cloud capabilities for HA - ie using multiple AZ's. Windows can be used for worker nodes, but not control plane.","title":"Provision underlying infrastructure to deploy a Kubernetes cluster"},{"location":"revision-topics/01-architcture-installation-configuration/#perform-a-version-upgrade-on-a-kubernetes-cluster-using-kubeadm","text":"First, install kubeadm to a specific version. This will determine the k8s version that it deploys: sudo apt-get update && sudo apt-get install -y kubeadm=1.19.0-00 kubelet=1.19.0-00 kubectl=1.19.0-00 && sudo apt-mark hold kubeadm Stand up a k8s cluster sudo kubeadm init --pod-network-cidr=10.244.0.0/16 Add CNI https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml To upgrade the underlying k8s cluster, we need to upgrade kubeadm. Update kubeadm sudo apt-mark unhold kubeadm sudo apt-get install --only-upgrade kubeadm Next we plan the upgrade - this won't change our cluster but will display what changes can be made: sudo kubeadm upgrade plan Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': COMPONENT CURRENT AVAILABLE kubelet 1 x v1.19.0 v1.20.2 Upgrade to the latest stable version: COMPONENT CURRENT AVAILABLE kube-apiserver v1.19.7 v1.20.2 kube-controller-manager v1.19.7 v1.20.2 kube-scheduler v1.19.7 v1.20.2 kube-proxy v1.19.7 v1.20.2 CoreDNS 1.7.0 1.7.0 etcd 3.4.9-1 3.4.13-0 You can now apply the upgrade by executing the following command: kubeadm upgrade apply v1.20.2 Important note: kubelet must be upgraded manually after this step. Upgrade the cluster: kubeadm upgrade apply v1.20.2 upgrade Kubelet: sudo apt-get install --only-upgrade kubelet kubectl","title":"Perform a version upgrade on a Kubernetes cluster using Kubeadm"},{"location":"revision-topics/01-architcture-installation-configuration/#implement-etcd-backup-and-restore","text":"","title":"Implement etcd backup and restore"},{"location":"revision-topics/01-architcture-installation-configuration/#backing-up-etcd","text":"Take a snapshot of the DB, then store it in a safe location: ETCDCTL_API=3 etcdctl snapshot save snapshot.db --cacert /etc/kubernetes/pki/etcd/server.crt --cert /etc/kubernetes/pki/etcd/ca.crt --key /etc/kubernetes/pki/etcd/ca.key Verify the backup: sudo ETCDCTL_API=3 etcdctl --write-out=table snapshot status snapshot.db +----------+----------+------------+------------+ | HASH | REVISION | TOTAL KEYS | TOTAL SIZE | +----------+----------+------------+------------+ | 2125d542 | 364069 | 770 | 3.8 MB | +----------+----------+------------+------------+","title":"Backing up etcd"},{"location":"revision-topics/01-architcture-installation-configuration/#restore-to-etcd","text":"To perform a restore: ETCDCTL_API=3 etcdctl snapshot restore snapshot.db","title":"Restore to etcd"},{"location":"revision-topics/02-workloads-and-scheduling/","text":"Workloads and Scheduling Understand deployments and how to perform rolling update and rollbacks Deployments are intended to replace Replication Controllers. They provide the same replication functions (through Replica Sets) and also the ability to rollout changes and roll them back if necessary. An example configuration is shown below: apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 5 template: metadata: labels: app: nginx-frontend spec: containers: - name: nginx image: nginx:1.14 ports: - containerPort: 80 The main reason why leverage deployments is to manage a number of identical pods via one administrative unit - the deployment object. Should we need to make changes, we apply this to the deployment object, not individual pods. Because of the declarative nature of deployments , Kubernetes will rectify any changes between desired and running state, and rectify accordingly. For example, if we manually deleted. We can then describe it with kubectl describe deployment nginx-deployment Update To update an existing deployment, we have two main options: Rolling Update Recreate A rolling update, as the name implies, will swap out containers in a deployment with one created by a new image. Use a rolling update when the application supports having a mix of different pods (aka application versions). This method will also involve no downtime of the service, but will take longer to bring up the deployment to the requested version. Old and new versions of the pod spec will coexist until they're all rotated. A recreation will delete all the existing pods and then spin up new ones. This method will involve downtime. Consider this a \u201cbing bang\u201d approach Examples listed in the Kubernetes documentation are largely imperative, but I prefer to be declarative. As an example, create a new yaml file and make the required changes, in this example, the version of the nginx container is incremented. apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 5 template: metadata: labels: app: nginx-frontend spec: containers: - name: nginx image: nginx:1.15 ports: - containerPort: 80 We can then apply this file kubectl apply -f updateddeployment.yaml --record=true Followed by the following: kubectl rollout status deployment/nginx-deployment Waiting for deployment \"nginx-deployment\" rollout to finish: 2 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 2 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 2 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 2 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 3 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 3 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 3 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 3 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 4 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 4 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 4 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 4 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 4 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 1 old replicas are pending termination... Waiting for deployment \"nginx-deployment\" rollout to finish: 1 old replicas are pending termination... Waiting for deployment \"nginx-deployment\" rollout to finish: 1 old replicas are pending termination... Waiting for deployment \"nginx-deployment\" rollout to finish: 4 of 5 updated replicas are available... deployment \"nginx-deployment\" successfully rolled out We can also use the kubectl rollout history to look at the revision history of a deployment kubectl rollout history deployment/nginx-deployment deployment.extensions/nginx-deployment REVISION CHANGE-CAUSE 1 <none> 2 <none> 4 <none> 5 kubectl apply --filename=updateddeployment.yaml --record=true Alternatively, we can also do this imperatively: kubectl --record deployments/nginx-deployment set image deployments/nginx-deployment nginx=nginx:1.9.1 deployment.extensions/nginx-deployment image updated deployment.extensions/nginx-deployment image updated Rollback To rollback to the previous version: kubectl rollout undo deployment/nginx-deployment To rollback to a specific version: kubectl rollout undo deployment/nginx-deployment --to-revision 5 Source of revision : kubectl rollout history deployment/nginx-deployment Use ConfigMaps and Secrets to configure applications Configmaps are a way to decouple configuration from a pod manifest. Obviously, the first step is to create a config map before we can get pods to use them: kubectl create configmap <map-name> <data-source> \u201cMap-name\u201d is an arbitrary name we give to this particular map, and \u201cdata-source\u201d corresponds to a key-value pair that resides in the config map. kubectl create configmap vt-cm --from-literal=blog=virtualthoughts.co.uk At which point we can then describe it: kubectl describe configmap vt-cm Name: vt-cm Namespace: default Labels: <none> Annotations: <none> Data ==== blog: ---- virtualthoughts.co.uk To reference this config map in a pod, we declare it in the respective yaml: Configmaps can be mounted as volumes or environment variables . The below example leverages the latter. apiVersion: v1 kind: Pod metadata: name: config-test-pod spec: containers: - name: test-container image: busybox command: [ \"/bin/sh\", \"-c\", \"env\" ] env: - name: BLOG_NAME valueFrom: configMapKeyRef: name: vt-cm key: blog restartPolicy: Never The pod above will output the environment variables, so we can validate it\u2019s leveraged the config map by extracting the logs from the pod: kubectl logs config-test-pod | grep \"BLOG_NAME=\" ... BLOG_NAME=virtualthoughts.co.uk ... Know how to scale applications Constantly adding more, individual pods is not a sustainable model for scaling an application. To facilitate applications at scale, we need to leverage higher level constructs such as replicasets or deployments. As mentioned previously, deployments provide us with a single administrative unit to manage the underlying pods. We can scale a deployment object to increase the number of pods . As an example, if the following is deployed: apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 5 template: metadata: labels: app: nginx-frontend spec: containers: - name: nginx image: nginx:1.14 ports: - containerPort: 80 If we wanted to scale this, we can simply modify the yaml file and scale up/down the deployment by modifying the \u201creplicas\u201d field, or modify it in the fly: kubectl scale deployment nginx-deployment --replicas 10 Understand the primitives used to create robust, self-healing, application deployments Deployments facilitate this by employing a reconciliation loop to check the number of deployed pods matches what\u2019s defined in the manifest. Under the hood, deployments leverage ReplicaSets, which are primarily responsible for this feature. Stateful Sets are similar to deployments, for example they manage the deployment and scaling of a series of pods. However, in addition to deployments they also provide guarantees about the ordering and uniqueness of Pods. A StatefulSet maintains a sticky identity for each of their Pods. These pods are created from the same spec, but are not interchangeable: each has a persistent identifier that it maintains across any rescheduling. StatefulSets are valuable for applications that require one or more of the following. Stable, unique network identifiers. Stable, persistent storage. Ordered, graceful deployment and scaling. Ordered, automated rolling updates. apiVersion: apps/v1 kind: StatefulSet metadata: name: nginx-statefulset spec: selector: matchLabels: app: vt-nginx serviceName: \"nginx\" replicas: 2 template: metadata: labels: app: vt-nginx spec: containers: - name: vt-nginx image: nginx:1.7.9 ports: - containerPort: 80 Understand how resource limits can affect Pod scheduling At a namespace level, we can define resource limits. This enables a restriction in resources, especially helpful in multi-tenancy environments and provides a mechanism to prevent pods from consuming more resources than permitted, which may have a detrimental effect on the environment as a whole. We can define the following: Default memory / CPU requests & limits for a namespace Minimum and Maximum memory / CPU constraints for a namespace Memory/CPU Quotas for a namespace Default Requests and Limits If a container is created in a namespace with a default request/limit value and doesn't explicitly define these in the manifest, it inherits these values from the namespace Note, if you define a container with a memory/CPU limit, but not a request, Kubernetes will define the limit the same as the request. Minimum / Maximum Constraints If a pod does not meet the range in which the constraints are valued at, it will not be scheduled. Quotas Control the total amount of CPU/memory that can be consumed in the namespace as a whole. Example: Attempt to schedule a pod that request more memory than defined in the namespace Create a namespace: kubectl create namespace tenant-mem-limited Create a YAML manifest to limit resources: apiVersion: v1 kind: LimitRange metadata: name: tenant-max-mem namespace: tenant-mem-limited spec: limits: - max: memory: 250Mi type: Container Apply this to the aforementioned namespace: kubectl apply -f maxmem.yaml To create a pod with a memory request that exceeds the limit: apiVersion: v1 kind: Pod metadata: name: too-much-memory namespace: tenant-mem-limited spec: containers: - name: too-much-mem image: nginx resources: requests: memory: \"300Mi\" Executing the above will yield the following result: The Pod \"too-much-memory\" is invalid: spec.containers[0].resources.requests: Invalid value: \"300Mi\": must be less than or equal to memory limit As we have defined the pod limit of the namespace to 250MiB, a request for 300MiB will fail. Awareness of manifest management and common templating tools Kustomize Kustomize is a templating tool for Kubernetes manifests in its native form (Yaml). When working with raw YAML files you will typically have a directory containing several files identifying the resources it creates. To begin, a directory containing our manifests needs to exist: /home/david/app/base total 16 drwxrwxr-x 2 david david 4096 Feb 9 11:44 . drwxr-xr-x 27 david david 4096 Feb 9 11:44 .. -rw-rw-r-- 1 david david 340 Feb 9 11:09 deployment.yaml -rw-rw-r-- 1 david david 153 Feb 9 11:09 service.yaml This will form our base - we will build on this but adding customisations in the form of overlays. First, we need a kustomize file. which can be created with kustomize create --autodetect This will create kustomization.yaml in the current directory: total 20 drwxrwxr-x 2 david david 4096 Feb 9 11:47 . drwxr-xr-x 27 david david 4096 Feb 9 11:47 .. -rw-rw-r-- 1 david david 340 Feb 9 11:09 deployment.yaml -rw-rw-r-- 1 david david 108 Feb 9 11:47 kustomization.yaml -rw-rw-r-- 1 david david 153 Feb 9 11:09 service.yaml The contents being: apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - deployment.yaml - service.yaml Variants and Overlays variant - Divergence in configuration from the base overlay - Composes variants together Say, for example, we wanted to generate manifests for different environments (prod and dev) that are based from this config, but have additional customisations. In this example we will create a dev variant encapsulated in a single Overlay mkdir -p overlays/{dev,prod} cd overlays/dev Begin by creating a Kustomization object specifying the base (this will create kustomization.yaml ) : kustomize create --resources ../../base In this example, I want to change the replica count to 1, as it's a dev environment. In the dev directory, create a new file deployment.yaml containing: apiVersion: apps/v1 kind: Deployment metadata: labels: app: nginx name: nginx-deployment spec: replicas: 1 The kustomization.yaml file needs modifying to include a patchesStrategicMerge block: apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization bases: - ../../base patchesStrategicMerge: - deployment.yaml Patches can be used to apply different customizations to Resources. Kustomize supports different patching mechanisms through patchesStrategicMerge and patchesJson6902 . patchesStrategicMerge is a list of file paths. We can generate the manifests and apply to the cluster by executing (from the base folder): kustomize build ./overlay/dev | kubectl apply -f - By running this, only 1 pod will be created in the deployment object, instead of what's defined in the base because of the customisation we've applied. We can do the same with prod, or any arbitrary number of environments. Helm Helm is synonymous to what apt or yum are in the Linux world. It's effectively a package manager for Kubernetes. \"Packages\" in Helm are called charts to which you can customise with your own values. It's unlikely the exam will require anyone to create a helm chart from scratch, but an understanding of how it works is a good idea. Helm Repos Repos are where helm charts are stored. Typically, a repo will contain a number of charts to choose from. Helm can be managed by a CLI client, and a repo can be added by running: helm repo add bitnami https://charts.bitnami.com/bitnami To list the packages from this repo: helm search repo bitnami To install a package from this repo: helm install my-release bitnami/mariadb Where my-release is a string identifying an installed instance of this application Parameters that can be customised - are dependent on how the chart is configured. For the aforementioned MariaDB chart, they are listed at https://github.com/bitnami/charts/tree/master/bitnami/mariadb/#parameters These values are encapsulated in the corresponding values.yaml file in the repo. You can populate an instance of it and apply it with: helm install -f https://raw.githubusercontent.com/bitnami/charts/master/bitnami/mariadb/values.yaml my-release bitnami/mariadb Alternatively, variables can be declared by using --set , such as: helm install my-release --set auth.rootPassword=secretpassword bitnami/mariadb","title":"Part Two - Workloads & Scheduling"},{"location":"revision-topics/02-workloads-and-scheduling/#workloads-and-scheduling","text":"","title":"Workloads and Scheduling"},{"location":"revision-topics/02-workloads-and-scheduling/#understand-deployments-and-how-to-perform-rolling-update-and-rollbacks","text":"Deployments are intended to replace Replication Controllers. They provide the same replication functions (through Replica Sets) and also the ability to rollout changes and roll them back if necessary. An example configuration is shown below: apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 5 template: metadata: labels: app: nginx-frontend spec: containers: - name: nginx image: nginx:1.14 ports: - containerPort: 80 The main reason why leverage deployments is to manage a number of identical pods via one administrative unit - the deployment object. Should we need to make changes, we apply this to the deployment object, not individual pods. Because of the declarative nature of deployments , Kubernetes will rectify any changes between desired and running state, and rectify accordingly. For example, if we manually deleted. We can then describe it with kubectl describe deployment nginx-deployment","title":"Understand deployments and how to perform rolling update and rollbacks"},{"location":"revision-topics/02-workloads-and-scheduling/#update","text":"To update an existing deployment, we have two main options: Rolling Update Recreate A rolling update, as the name implies, will swap out containers in a deployment with one created by a new image. Use a rolling update when the application supports having a mix of different pods (aka application versions). This method will also involve no downtime of the service, but will take longer to bring up the deployment to the requested version. Old and new versions of the pod spec will coexist until they're all rotated. A recreation will delete all the existing pods and then spin up new ones. This method will involve downtime. Consider this a \u201cbing bang\u201d approach Examples listed in the Kubernetes documentation are largely imperative, but I prefer to be declarative. As an example, create a new yaml file and make the required changes, in this example, the version of the nginx container is incremented. apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 5 template: metadata: labels: app: nginx-frontend spec: containers: - name: nginx image: nginx:1.15 ports: - containerPort: 80 We can then apply this file kubectl apply -f updateddeployment.yaml --record=true Followed by the following: kubectl rollout status deployment/nginx-deployment Waiting for deployment \"nginx-deployment\" rollout to finish: 2 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 2 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 2 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 2 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 3 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 3 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 3 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 3 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 4 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 4 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 4 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 4 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 4 out of 5 new replicas have been updated... Waiting for deployment \"nginx-deployment\" rollout to finish: 1 old replicas are pending termination... Waiting for deployment \"nginx-deployment\" rollout to finish: 1 old replicas are pending termination... Waiting for deployment \"nginx-deployment\" rollout to finish: 1 old replicas are pending termination... Waiting for deployment \"nginx-deployment\" rollout to finish: 4 of 5 updated replicas are available... deployment \"nginx-deployment\" successfully rolled out We can also use the kubectl rollout history to look at the revision history of a deployment kubectl rollout history deployment/nginx-deployment deployment.extensions/nginx-deployment REVISION CHANGE-CAUSE 1 <none> 2 <none> 4 <none> 5 kubectl apply --filename=updateddeployment.yaml --record=true Alternatively, we can also do this imperatively: kubectl --record deployments/nginx-deployment set image deployments/nginx-deployment nginx=nginx:1.9.1 deployment.extensions/nginx-deployment image updated deployment.extensions/nginx-deployment image updated","title":"Update"},{"location":"revision-topics/02-workloads-and-scheduling/#rollback","text":"To rollback to the previous version: kubectl rollout undo deployment/nginx-deployment To rollback to a specific version: kubectl rollout undo deployment/nginx-deployment --to-revision 5 Source of revision : kubectl rollout history deployment/nginx-deployment","title":"Rollback"},{"location":"revision-topics/02-workloads-and-scheduling/#use-configmaps-and-secrets-to-configure-applications","text":"Configmaps are a way to decouple configuration from a pod manifest. Obviously, the first step is to create a config map before we can get pods to use them: kubectl create configmap <map-name> <data-source> \u201cMap-name\u201d is an arbitrary name we give to this particular map, and \u201cdata-source\u201d corresponds to a key-value pair that resides in the config map. kubectl create configmap vt-cm --from-literal=blog=virtualthoughts.co.uk At which point we can then describe it: kubectl describe configmap vt-cm Name: vt-cm Namespace: default Labels: <none> Annotations: <none> Data ==== blog: ---- virtualthoughts.co.uk To reference this config map in a pod, we declare it in the respective yaml: Configmaps can be mounted as volumes or environment variables . The below example leverages the latter. apiVersion: v1 kind: Pod metadata: name: config-test-pod spec: containers: - name: test-container image: busybox command: [ \"/bin/sh\", \"-c\", \"env\" ] env: - name: BLOG_NAME valueFrom: configMapKeyRef: name: vt-cm key: blog restartPolicy: Never The pod above will output the environment variables, so we can validate it\u2019s leveraged the config map by extracting the logs from the pod: kubectl logs config-test-pod | grep \"BLOG_NAME=\" ... BLOG_NAME=virtualthoughts.co.uk ...","title":"Use ConfigMaps and Secrets to configure applications"},{"location":"revision-topics/02-workloads-and-scheduling/#know-how-to-scale-applications","text":"Constantly adding more, individual pods is not a sustainable model for scaling an application. To facilitate applications at scale, we need to leverage higher level constructs such as replicasets or deployments. As mentioned previously, deployments provide us with a single administrative unit to manage the underlying pods. We can scale a deployment object to increase the number of pods . As an example, if the following is deployed: apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 5 template: metadata: labels: app: nginx-frontend spec: containers: - name: nginx image: nginx:1.14 ports: - containerPort: 80 If we wanted to scale this, we can simply modify the yaml file and scale up/down the deployment by modifying the \u201creplicas\u201d field, or modify it in the fly: kubectl scale deployment nginx-deployment --replicas 10","title":"Know how to scale applications"},{"location":"revision-topics/02-workloads-and-scheduling/#understand-the-primitives-used-to-create-robust-self-healing-application-deployments","text":"Deployments facilitate this by employing a reconciliation loop to check the number of deployed pods matches what\u2019s defined in the manifest. Under the hood, deployments leverage ReplicaSets, which are primarily responsible for this feature. Stateful Sets are similar to deployments, for example they manage the deployment and scaling of a series of pods. However, in addition to deployments they also provide guarantees about the ordering and uniqueness of Pods. A StatefulSet maintains a sticky identity for each of their Pods. These pods are created from the same spec, but are not interchangeable: each has a persistent identifier that it maintains across any rescheduling. StatefulSets are valuable for applications that require one or more of the following. Stable, unique network identifiers. Stable, persistent storage. Ordered, graceful deployment and scaling. Ordered, automated rolling updates. apiVersion: apps/v1 kind: StatefulSet metadata: name: nginx-statefulset spec: selector: matchLabels: app: vt-nginx serviceName: \"nginx\" replicas: 2 template: metadata: labels: app: vt-nginx spec: containers: - name: vt-nginx image: nginx:1.7.9 ports: - containerPort: 80","title":"Understand the primitives used to create robust, self-healing, application deployments"},{"location":"revision-topics/02-workloads-and-scheduling/#understand-how-resource-limits-can-affect-pod-scheduling","text":"At a namespace level, we can define resource limits. This enables a restriction in resources, especially helpful in multi-tenancy environments and provides a mechanism to prevent pods from consuming more resources than permitted, which may have a detrimental effect on the environment as a whole. We can define the following: Default memory / CPU requests & limits for a namespace Minimum and Maximum memory / CPU constraints for a namespace Memory/CPU Quotas for a namespace","title":"Understand how resource limits can affect Pod scheduling"},{"location":"revision-topics/02-workloads-and-scheduling/#default-requests-and-limits","text":"If a container is created in a namespace with a default request/limit value and doesn't explicitly define these in the manifest, it inherits these values from the namespace Note, if you define a container with a memory/CPU limit, but not a request, Kubernetes will define the limit the same as the request.","title":"Default Requests and Limits"},{"location":"revision-topics/02-workloads-and-scheduling/#minimum-maximum-constraints","text":"If a pod does not meet the range in which the constraints are valued at, it will not be scheduled.","title":"Minimum / Maximum Constraints"},{"location":"revision-topics/02-workloads-and-scheduling/#quotas","text":"Control the total amount of CPU/memory that can be consumed in the namespace as a whole. Example: Attempt to schedule a pod that request more memory than defined in the namespace Create a namespace: kubectl create namespace tenant-mem-limited Create a YAML manifest to limit resources: apiVersion: v1 kind: LimitRange metadata: name: tenant-max-mem namespace: tenant-mem-limited spec: limits: - max: memory: 250Mi type: Container Apply this to the aforementioned namespace: kubectl apply -f maxmem.yaml To create a pod with a memory request that exceeds the limit: apiVersion: v1 kind: Pod metadata: name: too-much-memory namespace: tenant-mem-limited spec: containers: - name: too-much-mem image: nginx resources: requests: memory: \"300Mi\" Executing the above will yield the following result: The Pod \"too-much-memory\" is invalid: spec.containers[0].resources.requests: Invalid value: \"300Mi\": must be less than or equal to memory limit As we have defined the pod limit of the namespace to 250MiB, a request for 300MiB will fail.","title":"Quotas"},{"location":"revision-topics/02-workloads-and-scheduling/#awareness-of-manifest-management-and-common-templating-tools","text":"","title":"Awareness of manifest management and common templating tools"},{"location":"revision-topics/02-workloads-and-scheduling/#kustomize","text":"Kustomize is a templating tool for Kubernetes manifests in its native form (Yaml). When working with raw YAML files you will typically have a directory containing several files identifying the resources it creates. To begin, a directory containing our manifests needs to exist: /home/david/app/base total 16 drwxrwxr-x 2 david david 4096 Feb 9 11:44 . drwxr-xr-x 27 david david 4096 Feb 9 11:44 .. -rw-rw-r-- 1 david david 340 Feb 9 11:09 deployment.yaml -rw-rw-r-- 1 david david 153 Feb 9 11:09 service.yaml This will form our base - we will build on this but adding customisations in the form of overlays. First, we need a kustomize file. which can be created with kustomize create --autodetect This will create kustomization.yaml in the current directory: total 20 drwxrwxr-x 2 david david 4096 Feb 9 11:47 . drwxr-xr-x 27 david david 4096 Feb 9 11:47 .. -rw-rw-r-- 1 david david 340 Feb 9 11:09 deployment.yaml -rw-rw-r-- 1 david david 108 Feb 9 11:47 kustomization.yaml -rw-rw-r-- 1 david david 153 Feb 9 11:09 service.yaml The contents being: apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - deployment.yaml - service.yaml","title":"Kustomize"},{"location":"revision-topics/02-workloads-and-scheduling/#variants-and-overlays","text":"variant - Divergence in configuration from the base overlay - Composes variants together Say, for example, we wanted to generate manifests for different environments (prod and dev) that are based from this config, but have additional customisations. In this example we will create a dev variant encapsulated in a single Overlay mkdir -p overlays/{dev,prod} cd overlays/dev Begin by creating a Kustomization object specifying the base (this will create kustomization.yaml ) : kustomize create --resources ../../base In this example, I want to change the replica count to 1, as it's a dev environment. In the dev directory, create a new file deployment.yaml containing: apiVersion: apps/v1 kind: Deployment metadata: labels: app: nginx name: nginx-deployment spec: replicas: 1 The kustomization.yaml file needs modifying to include a patchesStrategicMerge block: apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization bases: - ../../base patchesStrategicMerge: - deployment.yaml Patches can be used to apply different customizations to Resources. Kustomize supports different patching mechanisms through patchesStrategicMerge and patchesJson6902 . patchesStrategicMerge is a list of file paths. We can generate the manifests and apply to the cluster by executing (from the base folder): kustomize build ./overlay/dev | kubectl apply -f - By running this, only 1 pod will be created in the deployment object, instead of what's defined in the base because of the customisation we've applied. We can do the same with prod, or any arbitrary number of environments.","title":"Variants and Overlays"},{"location":"revision-topics/02-workloads-and-scheduling/#helm","text":"Helm is synonymous to what apt or yum are in the Linux world. It's effectively a package manager for Kubernetes. \"Packages\" in Helm are called charts to which you can customise with your own values. It's unlikely the exam will require anyone to create a helm chart from scratch, but an understanding of how it works is a good idea.","title":"Helm"},{"location":"revision-topics/02-workloads-and-scheduling/#helm-repos","text":"Repos are where helm charts are stored. Typically, a repo will contain a number of charts to choose from. Helm can be managed by a CLI client, and a repo can be added by running: helm repo add bitnami https://charts.bitnami.com/bitnami To list the packages from this repo: helm search repo bitnami To install a package from this repo: helm install my-release bitnami/mariadb Where my-release is a string identifying an installed instance of this application Parameters that can be customised - are dependent on how the chart is configured. For the aforementioned MariaDB chart, they are listed at https://github.com/bitnami/charts/tree/master/bitnami/mariadb/#parameters These values are encapsulated in the corresponding values.yaml file in the repo. You can populate an instance of it and apply it with: helm install -f https://raw.githubusercontent.com/bitnami/charts/master/bitnami/mariadb/values.yaml my-release bitnami/mariadb Alternatively, variables can be declared by using --set , such as: helm install my-release --set auth.rootPassword=secretpassword bitnami/mariadb","title":"Helm Repos"},{"location":"revision-topics/03-services-and-networking/","text":"Services & Networking Understand host networking configuration on the cluster nodes At the host level, we have an interface (typically something like eth0 or ens192 etc) that acts as the primary network adapter. Each host is responsible for one subnet of the CNI range. In this example, the left host is responsible for 10.1.1.0/24, and the right host 10.1.2.0/24. The overall pod CIDR block may be something like 10.1.0.0/16. Virtual ethernet adapters are paired with a corresponding Pod network adapter. Kernel routing is used to enable Pods to communicate outside the host it resides in. Understand connectivity between Pods Every Pod gets its own IP address. This means you do not need to explicitly create links between Pods, and you almost never need to deal with mapping container ports to host ports. This creates a clean, backwards-compatible model where Pods can be treated much like VMs or physical hosts from the perspectives of port allocation, naming, service discovery, load balancing, application configuration, and migration. Kubernetes imposes the following fundamental requirements on any networking implementation (barring any intentional network segmentation policies): Pods on a node can communicate with all pods on all nodes without NAT Agents on a node (e.g. system daemons, Kubelet) can communicate with all pods on that node Note: When running workloads that leverage hostNetwork : Pods in the host network of a node can communicate with all pods on all nodes without NAT Understand ClusterIP, NodePort, LoadBalancer service types and endpoints Pods are ephemeral. Therefore, placing these behind a service which provides a stable, static entrypoint is a fundamental use of the kubernetes service object. To reiterate, services take the form of the following: ClusterIP - Internal only LoadBalancer - External, requires cloud provider, or software implementation to provide one NodePort - External, requires access the nodes directly Ingress resource - L7 An Ingress can be configured to give services externally-reachable URLs, load balance traffic, terminate SSL, and offer name based virtual hosting. An Ingress controller is responsible for fulfilling the Ingress, usually with a loadbalancer, though it may also configure your edge router or additional frontends to help handle the traffic. Know how to use Ingress controllers and Ingress resources Ingress exposes HTTP and HTTPS routes from outside the cluster to services within a cluster. Ingress consists of two components. Ingress Resource is a collection of rules for the inbound traffic to reach Services. These are Layer 7 (L7) rules that allow hostnames (and optionally paths) to be directed to specific Services in Kubernetes. The second component is the Ingress Controller which acts upon the rules set by the Ingress Resource, typically via an HTTP or L7 load balancer. It is vital that both pieces are properly configured to route traffic from an outside client to a Kubernetes Service. The following yaml creates two ingress rules for the website foo.bar.com The default path will direct traffic to the service \u201cdefault-service\u201d which listens on port 80 Paths ending in /foo will direct traffic to the service \u201cservice1\u201d which listens on port 4200 Paths ending in /bar will direct traffic to the service \u201cservice2\u201d which listens on port 8080 apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: simple-fanout-example spec: rules: - host: foo.bar.com http: paths: - path: / pathType: Prefix backend: service: name: default-service port: number: 80 - path: /foo pathType: Prefix backend: service: name: service1 port: number: 4200 - path: /bar pathType: Prefix backend: service: name: service2 port: number: 8080 In order for the Ingress resource to work, the cluster must have an ingress controller running. Ingress controllers are deployed into the Kubernetes cluster as a workload: > kubectl get po -A | grep nginx-ingress ingress-nginx nginx-ingress-controller-2gxtd 1/1 Running 0 14d ingress-nginx nginx-ingress-controller-9lrzh 1/1 Running 0 14d ingress-nginx nginx-ingress-controller-r2ksq 1/1 Running 0 14d Know how to configure and use CoreDNS As of 1.13, coredns has replaced kube-dns as the facilitator of cluster DNS and runs as pods. kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-fb8b8dccf-hxbhn 1/1 Running 9 10d coredns-fb8b8dccf-jks6g 1/1 Running 4 8d To view the DNS configuration of a pod, spin one up and inspect the /etc/resolv.conf: kubectl run busybox --image=busybox -- sleep 9000 kubectl exec -it busybox sh / # cat /etc/resolv.conf nameserver 10.96.0.10 search default.svc.cluster.local svc.cluster.local cluster.local virtualthoughts.co.uk options ndots:5 \u201c10.96.0.10\u201d references the Kube-DNS service \u201cDefault.svc.cluster.local\u201d References the namespace with the suffix svc.cluster.local. All pods are provisioned a DNS record and are in the format of [Pod IP separated by dashes].[Namespace].[type].[Base Domain Name] Where [type] is pod in this example, put services can be resolved by the same convention. For example: / # nslookup 10-42-2-68.default.pod.cluster.local Server: 10.43.0.10 Address: 10.43.0.10:53 Name: 10-42-2-68.default.pod.cluster.local Address: 10.42.2.68 Services follow a similar pattern [Service Name].[Namespace].[type].[Base Domain Name] For example: my-svc.my-namespace.svc.cluster-domain.example Headless services are those without a cluster ip, but will respond with a list of IP\u2019s of pods that are applicable at that particular moment in time. apiVersion: v1 kind: Service metadata: name: test-headless spec: clusterIP: None ports: - port: 80 targetPort: 80 selector: app: web-headless We can modify the default behavior of the pod dns configuration in the yaml file: apiVersion: v1 kind: Pod metadata: namespace: default name: dns-example spec: containers: - name: test image: nginx dnsPolicy: \"None\" dnsConfig: nameservers: - 8.8.8.8 searches: - ns1.svc.cluster.local - my.dns.search.suffix options: - name: ndots value: \"2\" - name: edns0 CoreDNS also has a configmap that can be modified: kubectl get cm coredns -n kube-system -o yaml apiVersion: v1 data: Corefile: | .:53 { errors health ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure upstream fallthrough in-addr.arpa ip6.arpa } hosts /etc/coredns/NodeHosts { reload 1s fallthrough } prometheus :9153 forward . /etc/resolv.conf cache 30 loop reload loadbalance } NodeHosts: | 172.16.10.100 k3s-ranch-node-1 172.16.10.101 k3s-ranch-node-2 172.16.10.102 k3s-ranch-node-3 The Corefile configuration includes the following plugins of CoreDNS: errors : Errors are logged to stdout. health : Health of CoreDNS is reported to http://localhost:8080/health . In this extended syntax lameduck will make the process unhealthy then wait for 5 seconds before the process is shut down. ready : An HTTP endpoint on port 8181 will return 200 OK, when all plugins that are able to signal readiness have done so. kubernetes : CoreDNS will reply to DNS queries based on IP of the services and pods of Kubernetes. You can find more details about that plugin on the CoreDNS website. ttl allows you to set a custom TTL for responses. The default is 5 seconds. The minimum TTL allowed is 0 seconds, and the maximum is capped at 3600 seconds. Setting TTL to 0 will prevent records from being cached. The pods insecure option is provided for backward compatibility with kube-dns. You can use the pods verified option, which returns an A record only if there exists a pod in same namespace with matching IP. The pods disabled option can be used if you don't use pod records. prometheus : Metrics of CoreDNS are available at http://localhost:9153/metrics in Prometheus format (also known as OpenMetrics). forward : Any queries that are not within the cluster domain of Kubernetes will be forwarded to predefined resolvers (/etc/resolv.conf). cache: This enables a frontend cache. loop : Detects simple forwarding loops and halts the CoreDNS process if a loop is found. reload : Allows automatic reload of a changed Corefile. After you edit the ConfigMap configuration, allow two minutes for your changes to take effect. You can modify the default CoreDNS behavior by modifying the ConfigMap. Choose an appropriate container network interface plugin You must deploy a Container Network Interface (CNI) based Pod network add-on so that your Pods can communicate with each other. Cluster DNS (CoreDNS) will not start up before a network is installed. https://kubernetes.io/docs/concepts/cluster-administration/addons/#networking-and-network-policy As a generalisation, CNI\u2019s provide some kind of network overlay. But each have their own features, limitations and considerations. CNI's manifest and Kubernetes pods in your cluster. A typical workflow would involve standing up your k8s cluster and applying a network cni with: kubectl apply -f <add-on.yaml>","title":"Part Three - Services & Networking"},{"location":"revision-topics/03-services-and-networking/#services-networking","text":"","title":"Services &amp; Networking"},{"location":"revision-topics/03-services-and-networking/#understand-host-networking-configuration-on-the-cluster-nodes","text":"At the host level, we have an interface (typically something like eth0 or ens192 etc) that acts as the primary network adapter. Each host is responsible for one subnet of the CNI range. In this example, the left host is responsible for 10.1.1.0/24, and the right host 10.1.2.0/24. The overall pod CIDR block may be something like 10.1.0.0/16. Virtual ethernet adapters are paired with a corresponding Pod network adapter. Kernel routing is used to enable Pods to communicate outside the host it resides in.","title":"Understand host networking configuration on the cluster nodes"},{"location":"revision-topics/03-services-and-networking/#understand-connectivity-between-pods","text":"Every Pod gets its own IP address. This means you do not need to explicitly create links between Pods, and you almost never need to deal with mapping container ports to host ports. This creates a clean, backwards-compatible model where Pods can be treated much like VMs or physical hosts from the perspectives of port allocation, naming, service discovery, load balancing, application configuration, and migration. Kubernetes imposes the following fundamental requirements on any networking implementation (barring any intentional network segmentation policies): Pods on a node can communicate with all pods on all nodes without NAT Agents on a node (e.g. system daemons, Kubelet) can communicate with all pods on that node Note: When running workloads that leverage hostNetwork : Pods in the host network of a node can communicate with all pods on all nodes without NAT","title":"Understand connectivity between Pods"},{"location":"revision-topics/03-services-and-networking/#understand-clusterip-nodeport-loadbalancer-service-types-and-endpoints","text":"Pods are ephemeral. Therefore, placing these behind a service which provides a stable, static entrypoint is a fundamental use of the kubernetes service object. To reiterate, services take the form of the following: ClusterIP - Internal only LoadBalancer - External, requires cloud provider, or software implementation to provide one NodePort - External, requires access the nodes directly Ingress resource - L7 An Ingress can be configured to give services externally-reachable URLs, load balance traffic, terminate SSL, and offer name based virtual hosting. An Ingress controller is responsible for fulfilling the Ingress, usually with a loadbalancer, though it may also configure your edge router or additional frontends to help handle the traffic.","title":"Understand ClusterIP, NodePort, LoadBalancer service types and endpoints"},{"location":"revision-topics/03-services-and-networking/#know-how-to-use-ingress-controllers-and-ingress-resources","text":"Ingress exposes HTTP and HTTPS routes from outside the cluster to services within a cluster. Ingress consists of two components. Ingress Resource is a collection of rules for the inbound traffic to reach Services. These are Layer 7 (L7) rules that allow hostnames (and optionally paths) to be directed to specific Services in Kubernetes. The second component is the Ingress Controller which acts upon the rules set by the Ingress Resource, typically via an HTTP or L7 load balancer. It is vital that both pieces are properly configured to route traffic from an outside client to a Kubernetes Service. The following yaml creates two ingress rules for the website foo.bar.com The default path will direct traffic to the service \u201cdefault-service\u201d which listens on port 80 Paths ending in /foo will direct traffic to the service \u201cservice1\u201d which listens on port 4200 Paths ending in /bar will direct traffic to the service \u201cservice2\u201d which listens on port 8080 apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: simple-fanout-example spec: rules: - host: foo.bar.com http: paths: - path: / pathType: Prefix backend: service: name: default-service port: number: 80 - path: /foo pathType: Prefix backend: service: name: service1 port: number: 4200 - path: /bar pathType: Prefix backend: service: name: service2 port: number: 8080 In order for the Ingress resource to work, the cluster must have an ingress controller running. Ingress controllers are deployed into the Kubernetes cluster as a workload: > kubectl get po -A | grep nginx-ingress ingress-nginx nginx-ingress-controller-2gxtd 1/1 Running 0 14d ingress-nginx nginx-ingress-controller-9lrzh 1/1 Running 0 14d ingress-nginx nginx-ingress-controller-r2ksq 1/1 Running 0 14d","title":"Know how to use Ingress controllers and Ingress resources"},{"location":"revision-topics/03-services-and-networking/#know-how-to-configure-and-use-coredns","text":"As of 1.13, coredns has replaced kube-dns as the facilitator of cluster DNS and runs as pods. kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-fb8b8dccf-hxbhn 1/1 Running 9 10d coredns-fb8b8dccf-jks6g 1/1 Running 4 8d To view the DNS configuration of a pod, spin one up and inspect the /etc/resolv.conf: kubectl run busybox --image=busybox -- sleep 9000 kubectl exec -it busybox sh / # cat /etc/resolv.conf nameserver 10.96.0.10 search default.svc.cluster.local svc.cluster.local cluster.local virtualthoughts.co.uk options ndots:5 \u201c10.96.0.10\u201d references the Kube-DNS service \u201cDefault.svc.cluster.local\u201d References the namespace with the suffix svc.cluster.local. All pods are provisioned a DNS record and are in the format of [Pod IP separated by dashes].[Namespace].[type].[Base Domain Name] Where [type] is pod in this example, put services can be resolved by the same convention. For example: / # nslookup 10-42-2-68.default.pod.cluster.local Server: 10.43.0.10 Address: 10.43.0.10:53 Name: 10-42-2-68.default.pod.cluster.local Address: 10.42.2.68 Services follow a similar pattern [Service Name].[Namespace].[type].[Base Domain Name] For example: my-svc.my-namespace.svc.cluster-domain.example Headless services are those without a cluster ip, but will respond with a list of IP\u2019s of pods that are applicable at that particular moment in time. apiVersion: v1 kind: Service metadata: name: test-headless spec: clusterIP: None ports: - port: 80 targetPort: 80 selector: app: web-headless We can modify the default behavior of the pod dns configuration in the yaml file: apiVersion: v1 kind: Pod metadata: namespace: default name: dns-example spec: containers: - name: test image: nginx dnsPolicy: \"None\" dnsConfig: nameservers: - 8.8.8.8 searches: - ns1.svc.cluster.local - my.dns.search.suffix options: - name: ndots value: \"2\" - name: edns0 CoreDNS also has a configmap that can be modified: kubectl get cm coredns -n kube-system -o yaml apiVersion: v1 data: Corefile: | .:53 { errors health ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure upstream fallthrough in-addr.arpa ip6.arpa } hosts /etc/coredns/NodeHosts { reload 1s fallthrough } prometheus :9153 forward . /etc/resolv.conf cache 30 loop reload loadbalance } NodeHosts: | 172.16.10.100 k3s-ranch-node-1 172.16.10.101 k3s-ranch-node-2 172.16.10.102 k3s-ranch-node-3 The Corefile configuration includes the following plugins of CoreDNS: errors : Errors are logged to stdout. health : Health of CoreDNS is reported to http://localhost:8080/health . In this extended syntax lameduck will make the process unhealthy then wait for 5 seconds before the process is shut down. ready : An HTTP endpoint on port 8181 will return 200 OK, when all plugins that are able to signal readiness have done so. kubernetes : CoreDNS will reply to DNS queries based on IP of the services and pods of Kubernetes. You can find more details about that plugin on the CoreDNS website. ttl allows you to set a custom TTL for responses. The default is 5 seconds. The minimum TTL allowed is 0 seconds, and the maximum is capped at 3600 seconds. Setting TTL to 0 will prevent records from being cached. The pods insecure option is provided for backward compatibility with kube-dns. You can use the pods verified option, which returns an A record only if there exists a pod in same namespace with matching IP. The pods disabled option can be used if you don't use pod records. prometheus : Metrics of CoreDNS are available at http://localhost:9153/metrics in Prometheus format (also known as OpenMetrics). forward : Any queries that are not within the cluster domain of Kubernetes will be forwarded to predefined resolvers (/etc/resolv.conf). cache: This enables a frontend cache. loop : Detects simple forwarding loops and halts the CoreDNS process if a loop is found. reload : Allows automatic reload of a changed Corefile. After you edit the ConfigMap configuration, allow two minutes for your changes to take effect. You can modify the default CoreDNS behavior by modifying the ConfigMap.","title":"Know how to configure and use CoreDNS"},{"location":"revision-topics/03-services-and-networking/#choose-an-appropriate-container-network-interface-plugin","text":"You must deploy a Container Network Interface (CNI) based Pod network add-on so that your Pods can communicate with each other. Cluster DNS (CoreDNS) will not start up before a network is installed. https://kubernetes.io/docs/concepts/cluster-administration/addons/#networking-and-network-policy As a generalisation, CNI\u2019s provide some kind of network overlay. But each have their own features, limitations and considerations. CNI's manifest and Kubernetes pods in your cluster. A typical workflow would involve standing up your k8s cluster and applying a network cni with: kubectl apply -f <add-on.yaml>","title":"Choose an appropriate container network interface plugin"},{"location":"revision-topics/04-storage/","text":"Storage Understand storage classes, persistent volumes A StorageClass provides a way for administrators to describe the \"classes\" of storage they offer. Different classes might map to quality-of-service levels, or to backup policies, or to arbitrary policies determined by the cluster administrators. Kubernetes itself is un-opinionated about what classes represent. This concept is sometimes called \"profiles\" in other storage systems. An example of a storage class is below: apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: standard provisioner: kubernetes.io/aws-ebs parameters: type: gp2 reclaimPolicy: Retain allowVolumeExpansion: true mountOptions: - debug volumeBindingMode: Immediate Key declarations are: provisioner : Determine which volume plugin to use. This usually matches to a cloud provider, and a specific storage service that it offers. In this example AWS Elastic Block Store parameters : Describe characteristics of this storage class in context of the underlying provisioner. In this example, the type is gp2 which, in AWS lingo means General Purpose SSD. Other types include IO1 (Provisioned IOPS), ST1 (Throughput Optimised) and STC (Cold Storage). A storageclass object by itself defines what storage is provisioned when it is invoked . By itself, it does nothing. A persistentvolume object can be used to request storage from a storageclass and is typically part of a pod manifest apiVersion: v1 kind: PersistentVolume metadata: name: pv0003 spec: capacity: storage: 5Gi volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Recycle storageClassName: standard Understand volume mode, access modes and reclaim policies for volumes Volume modes Only two exist: block - Mounted to a pod as a raw block device without a filesystem. The Pod / application needs to understand how to deal with raw block devices. Presenting it in this way can yield better performance, at the expense of complexity. filesystem - Mounted inside a pods' filesystem inside a directory. If the volume is backed by a block device with no filesystem, Kubernetes will create one. Compared to block devices, this method offers the highest compatibility, at the expense of performance. Access Modes Three options exist: ReadWriteOnce \u2013 The volume can be mounted as read-write by a single node ReadOnlyMany \u2013 The volume can be mounted read-only by many nodes ReadWriteMany \u2013 The volume can be mounted as read-write by many nodes Understand persistent volume claims primitive A PersistentVolume can be thought of as storage provisioned by an administrator. Think of this as pre-allocation A PersistentVolumeClaim can be thought of as storage requested by a user/workload. When the user creates PVC to request storage, Kubernetes will try to match that PVC with a pre-allocated PV. If a match can be found, the PVC will be bound to the PV, and the user will start to use that pre-allocated piece of storage. Know how to configure applications with persistent storage When not using storageclasses there are a number of steps: Create the PV: apiVersion: v1 kind: PersistentVolume metadata: name: task-pv-volume labels: type: local spec: storageClassName: manual capacity: storage: 10Gi accessModes: - ReadWriteOnce hostPath: path: \"/mnt/data\" Create the workload to leverage this: apiVersion: v1 kind: PersistentVolumeClaim metadata: name: task-pv-claim spec: storageClassName: manual accessModes: - ReadWriteOnce resources: requests: storage: 3Gi --- apiVersion: v1 kind: Pod metadata: name: task-pv-pod spec: volumes: - name: task-pv-storage persistentVolumeClaim: claimName: task-pv-claim containers: - name: task-pv-container image: nginx ports: - containerPort: 80 name: \"http-server\" volumeMounts: - mountPath: \"/usr/share/nginx/html\" name: task-pv-storage If leveraging a storageclass, a persistentvolume object is not required, we just need a persistentvolumeclaim apiVersion: v1 kind: PersistentVolumeClaim metadata: name: task-pv-claim spec: storageClassName: myStorageClass accessModes: - ReadWriteOnce resources: requests: storage: 3Gi --- apiVersion: v1 kind: Pod metadata: name: task-pv-pod spec: volumes: - name: task-pv-storage persistentVolumeClaim: claimName: task-pv-claim containers: - name: task-pv-container image: nginx ports: - containerPort: 80 name: \"http-server\" volumeMounts: - mountPath: \"/usr/share/nginx/html\" name: task-pv-storage","title":"Part Four - Storage"},{"location":"revision-topics/04-storage/#storage","text":"","title":"Storage"},{"location":"revision-topics/04-storage/#understand-storage-classes-persistent-volumes","text":"A StorageClass provides a way for administrators to describe the \"classes\" of storage they offer. Different classes might map to quality-of-service levels, or to backup policies, or to arbitrary policies determined by the cluster administrators. Kubernetes itself is un-opinionated about what classes represent. This concept is sometimes called \"profiles\" in other storage systems. An example of a storage class is below: apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: standard provisioner: kubernetes.io/aws-ebs parameters: type: gp2 reclaimPolicy: Retain allowVolumeExpansion: true mountOptions: - debug volumeBindingMode: Immediate Key declarations are: provisioner : Determine which volume plugin to use. This usually matches to a cloud provider, and a specific storage service that it offers. In this example AWS Elastic Block Store parameters : Describe characteristics of this storage class in context of the underlying provisioner. In this example, the type is gp2 which, in AWS lingo means General Purpose SSD. Other types include IO1 (Provisioned IOPS), ST1 (Throughput Optimised) and STC (Cold Storage). A storageclass object by itself defines what storage is provisioned when it is invoked . By itself, it does nothing. A persistentvolume object can be used to request storage from a storageclass and is typically part of a pod manifest apiVersion: v1 kind: PersistentVolume metadata: name: pv0003 spec: capacity: storage: 5Gi volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Recycle storageClassName: standard","title":"Understand storage classes, persistent volumes"},{"location":"revision-topics/04-storage/#understand-volume-mode-access-modes-and-reclaim-policies-for-volumes","text":"","title":"Understand volume mode, access modes and reclaim policies for volumes"},{"location":"revision-topics/04-storage/#volume-modes","text":"Only two exist: block - Mounted to a pod as a raw block device without a filesystem. The Pod / application needs to understand how to deal with raw block devices. Presenting it in this way can yield better performance, at the expense of complexity. filesystem - Mounted inside a pods' filesystem inside a directory. If the volume is backed by a block device with no filesystem, Kubernetes will create one. Compared to block devices, this method offers the highest compatibility, at the expense of performance.","title":"Volume modes"},{"location":"revision-topics/04-storage/#access-modes","text":"Three options exist: ReadWriteOnce \u2013 The volume can be mounted as read-write by a single node ReadOnlyMany \u2013 The volume can be mounted read-only by many nodes ReadWriteMany \u2013 The volume can be mounted as read-write by many nodes","title":"Access Modes"},{"location":"revision-topics/04-storage/#understand-persistent-volume-claims-primitive","text":"A PersistentVolume can be thought of as storage provisioned by an administrator. Think of this as pre-allocation A PersistentVolumeClaim can be thought of as storage requested by a user/workload. When the user creates PVC to request storage, Kubernetes will try to match that PVC with a pre-allocated PV. If a match can be found, the PVC will be bound to the PV, and the user will start to use that pre-allocated piece of storage.","title":"Understand persistent volume claims primitive"},{"location":"revision-topics/04-storage/#know-how-to-configure-applications-with-persistent-storage","text":"When not using storageclasses there are a number of steps: Create the PV: apiVersion: v1 kind: PersistentVolume metadata: name: task-pv-volume labels: type: local spec: storageClassName: manual capacity: storage: 10Gi accessModes: - ReadWriteOnce hostPath: path: \"/mnt/data\" Create the workload to leverage this: apiVersion: v1 kind: PersistentVolumeClaim metadata: name: task-pv-claim spec: storageClassName: manual accessModes: - ReadWriteOnce resources: requests: storage: 3Gi --- apiVersion: v1 kind: Pod metadata: name: task-pv-pod spec: volumes: - name: task-pv-storage persistentVolumeClaim: claimName: task-pv-claim containers: - name: task-pv-container image: nginx ports: - containerPort: 80 name: \"http-server\" volumeMounts: - mountPath: \"/usr/share/nginx/html\" name: task-pv-storage If leveraging a storageclass, a persistentvolume object is not required, we just need a persistentvolumeclaim apiVersion: v1 kind: PersistentVolumeClaim metadata: name: task-pv-claim spec: storageClassName: myStorageClass accessModes: - ReadWriteOnce resources: requests: storage: 3Gi --- apiVersion: v1 kind: Pod metadata: name: task-pv-pod spec: volumes: - name: task-pv-storage persistentVolumeClaim: claimName: task-pv-claim containers: - name: task-pv-container image: nginx ports: - containerPort: 80 name: \"http-server\" volumeMounts: - mountPath: \"/usr/share/nginx/html\" name: task-pv-storage","title":"Know how to configure applications with persistent storage"},{"location":"revision-topics/05-troubleshooting/","text":"Troubleshooting Evaluate cluster and node logging Master Node(s) ETCD Usually, most etcd implementations also include etcdctl, which can aid in monitoring the state of the cluster. If you\u2019re unsure where to find it, execute the following: find / -name etcdctl Leveraging this tool to check the cluster status: etcdctl --write-out=table --endpoints=$ENDPOINTS endpoint status +------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | https://127.0.0.1:2379 | 4e30a295f2c3c1a4 | 3.5.0 | 8.1 MB | true | false | 3 | 7903 | 7903 | | +------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ The cluster this was executed on has only one master node, hence only one result from the script. You will normally receive a response for each etcd member in the cluster. Alternatively, leverage kubectl get componentstatuses: kubectl get componentstatuses #ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok etcd-1 Healthy {\"health\":\"true\"} etcd-0 Healthy {\"health\":\"true\"} Etcd may also be running as a Pod: kubectl logs etcd-ubuntu -n kube-system Kube-apiserver This is dependent on the environment for which the Kubernetes platform has been installed on. For systemd based systems: journalctl -u kube-apiserver Or cat /var/log/kube-apiserver.log Or for instances where Kube-API server is running as a static pod: kubectl logs kube-apiserver-k8s-master-03 -n kube-system Kube-Scheduler For systemd-based systems journalctl -u kube-scheduler Or cat /var/log/kube-scheduler.log Or for instances where Kube-Scheduler is running as a static pod: kubectl logs kube-scheduler-k8s-master-03 -n kube-system Kube-Controller-Manager For systemd-based systems journalctl -u kube-controller-manager Or cat /var/log/kube-controller-manager.log Or for instances where Kube-controller manager is running as a static pod: kubectl logs kube-controller-manager-k8s-master-03 -n kube-system Worker Node(s) CNI Obviously this is dependent on the CNI in use for the cluster you\u2019re working on. However, using Flannel as an example: journalctl -u flanneld If running as a pod, however: Kubectl logs --namespace kube-system <POD-ID> -c kube-flannel kubectl logs --namespace kube-system weave-net-pwjkj -c weave Kube-Proxy For systemd-based systems journalctl -u kube-proxy Or cat /var/log/kube-proxy.log Or for instances where Kube-proxy manager is running as a static pod: kubectl logs kube-proxy -n kube-system Kubelet journalctl -u kubelet Or cat /var/log/kubelet.log Container Runtime Similarly to the CNI, this depends on which container runtime has been deployed, but using Docker as an example: For systemd-based systems: journalctl -u docker.service Or cat /var/log/docker.log Hint : list the contents of etc/systemd/system if it\u2019s a systemd-based service (containerd.service may be here) Cluster Logging At a cluster level, kubectl get events provides a good overview. Understand how to monitor applications This section is a bit open-ended as it highly depends on what you have deployed and the topology of an application. Typically, however, we have an application that runs as a number of inter-connected microservices , consequently we monitor our applications by monitoring the underlying objects that comprise it, such as: Pods Deployments Services etc Manage container stdout & stderr logs Kubernetes handles and redirects any output generated from a containers stdout and stderr streams. These get directed through a logging driver which influences where to store these logs. Different implementations of Docker differ in exact implementation (such as RHEL's flavor of Docker) but commonly, these drivers will write to a file in json format: root@ubuntu:~# docker info | grep \"Logging Driver\" Logging Driver: json-file The location for these logs is typically /var/log/containers but can be tweaked. Additionally, these contain symlinks: root@ubuntu:~# ls -la /var/log/containers/ total 44 drwxr-xr-x 2 root root 4096 Feb 8 19:18 . drwxrwxr-x 11 root syslog 4096 Feb 12 00:00 .. lrwxrwxrwx 1 root root 100 Feb 8 19:17 coredns-74ff55c5b-j4trd_kube-system_coredns-5d65324791ffcdf45d3552d875c6834f9a305c5be84b18745cb1657f784e5dd0.log -> /var/log/pods/kube-system_coredns-74ff55c5b-j4trd_4afbef57-5592-4edb-96af-9d17f595d160/coredns/0.log lrwxrwxrwx 1 root root 100 Feb 8 19:17 coredns-74ff55c5b-wrgkr_kube-system_coredns-b2fcfa679e9725dbe601bc1a0f218121a9c44b91d7300bbb57039a4edd219991.log -> /var/log/pods/kube-system_coredns-74ff55c5b-wrgkr_b64ac6d5-654b-4194-b6a8-5f8aa4c3cbe2/coredns/0.log lrwxrwxrwx 1 root root 81 Feb 8 19:16 etcd-ubuntu_kube-system_etcd-fcc5bc99932f380781776baa125b6f3be035e18fcec520afb827102e2afce1cd.log -> /var/log/pods/kube-system_etcd-ubuntu_f608198a8b73b3cf090bd15e2823df04/etcd/0.log lrwxrwxrwx 1 root root 101 Feb 8 19:16 kube-apiserver-ubuntu_kube-system_kube-apiserver-a264bbd54b7f23c8d424b0b368a48fdd1c5dcecc72fca95a460c146b2b5d85f5.log -> /var/log/pods/kube-system_kube-apiserver-ubuntu_212641053a16fa2bb404ccde20f6eaf0/kube-apiserver/0.log lrwxrwxrwx 1 root root 119 Feb 8 19:17 kube-controller-manager-ubuntu_kube-system_kube-controller-manager-a4ef7fe2b52272ea77f8de2da0989a9bcee757ae778fc08f1786b26b45bf13e1.log -> /var/log/pods/kube-system_kube-controller-manager-ubuntu_7bbe7d37f1b2c7586237165580c2f5c3/kube-controller-manager/0.log lrwxrwxrwx 1 root root 102 Feb 8 19:05 kube-flannel-ds-rfsfs_kube-system_install-cni-f8762e22fbf17925432682bdb1259a066208c62fa695d09cd6ee9b0cef3d36ba.log -> /var/log/pods/kube-system_kube-flannel-ds-rfsfs_2892d4e3-e326-4b4b-90c0-396fb80863ca/install-cni/0.log lrwxrwxrwx 1 root root 103 Feb 8 19:05 kube-flannel-ds-rfsfs_kube-system_kube-flannel-f96e019717814d7360e1aacd275cac121c13e0ee94cc5c93dcb35365608e6f83.log -> /var/log/pods/kube-system_kube-flannel-ds-rfsfs_2892d4e3-e326-4b4b-90c0-396fb80863ca/kube-flannel/0.log lrwxrwxrwx 1 root root 96 Feb 8 19:17 kube-proxy-l52f9_kube-system_kube-proxy-bfe08cb8663b46551e8608c094194ec61d03edfa7d25a6f414c07ed6563ada89.log -> /var/log/pods/kube-system_kube-proxy-l52f9_d2b73ed1-5df4-4a18-9595-20798db4f110/kube-proxy/0.log lrwxrwxrwx 1 root root 101 Feb 8 19:17 kube-scheduler-ubuntu_kube-system_kube-scheduler-4a695e53684f4591ec9385d6944f7841c0329aa49be220e5af6304da281cb41a.log -> /var/log/pods/kube-system_kube-scheduler-ubuntu_69cd289b4ed80ced4f95a59ff60fa102/kube-scheduler/0.log Troubleshoot application failure This is a somewhat ambitious topic to cover as how we approach troubleshooting application failures varies by the architecture of that application, which resources/API objects we're leveraging, if the application contains logs. However, good starting points would include running things like: kubectl describe <object> kubectl logs <podname> kubectl get events Troubleshoot cluster component failure Covered in \"Evaluate cluster and node logging\" Troubleshoot networking DNS Resolution Pods and Services will automatically have a DNS record registered against coredns in the cluster, aka \"A\" records for IPv4 and \"AAAA\" for IPv6. The format of which is: pod-ip-address.my-namespace.pod.cluster-domain.example my-svc-name.my-namespace.svc.cluster-domain.example Pod DNS records resolve to a single entity, even if the Pod contains multiple containers as they share the same networking space. Service DNS records resolve to the respective service object. Pods will automatically have their DNS resolution configured based on coredns settings. This can be validated by opening a shell to the pod and inspecting /etc/resolv.conf: > kubectl exec -it web-server sh kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead. / # cat /etc/resolv.conf nameserver 10.43.0.10 search default.svc.cluster.local svc.cluster.local cluster.local eu-central-1.compute.internal options ndots:5 10.43.0.10 being the coredns service object: > kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 16d To test resolution, we can run a pod with nslookup to test. For the pod below: > kubectl get po -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES web-server 1/1 Running 0 2d20h 10.42.1.31 ip-172-31-36-67 <none> <none> Knowing the format of the A record: pod-ip-address.my-namespace.pod.cluster-domain.example We should be able to resolve 10-42-1-31.default.pod.cluster.local . Tip : To determine the cluster domain, inspect the coredns configmap. Below indicating cluster.local . > kubectl get cm coredns -n kube-system -o yaml apiVersion: v1 data: Corefile: | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { Create a Pod with the tools required: kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml Test lookup: kubectl exec -i -t dnsutils -- nslookup 10-42-1-31.default.pod.cluster.local > kubectl exec -i -t dnsutils -- nslookup 10-42-1-31.default.pod.cluster.local Server: 10.43.0.10 Address: 10.43.0.10#53 Name: 10-42-1-31.default.pod.cluster.local Address: 10.42.1.31 Similarly, for a service, in this case a service called nginx-service that resides in the default namespace: > kubectl exec -i -t dnsutils -- nslookup nginx-service.default.svc.cluster.local Server: 10.43.0.10 Address: 10.43.0.10#53 Name: nginx-service.default.svc.cluster.local Address: 10.43.0.223 > kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx-service ClusterIP 10.43.0.223 <none> 80/TCP 9m15s CNI Issues Mainly covered earlier in acquiring logs for the CNI. However, one issue that might occur is when a CNI is incorrectly, or not initialised. This may cause workloads to enter a pending status: kubectl get po -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx 0/1 Pending 0 57s <none> <none> <none> <none> kubectl describe <pod> can help identify issues with assigning IP addresses to nodes from the CNI Port Checking Similarly, with leveraging nslookup to validate DNS resolution in our cluster, we can lean on other tools to perform other diagnostic. All we need is a pod that has a utility like netcat , telnet etc.","title":"Part Five - Troubleshooting"},{"location":"revision-topics/05-troubleshooting/#troubleshooting","text":"","title":"Troubleshooting"},{"location":"revision-topics/05-troubleshooting/#evaluate-cluster-and-node-logging","text":"","title":"Evaluate cluster and node logging"},{"location":"revision-topics/05-troubleshooting/#master-nodes","text":"","title":"Master Node(s)"},{"location":"revision-topics/05-troubleshooting/#etcd","text":"Usually, most etcd implementations also include etcdctl, which can aid in monitoring the state of the cluster. If you\u2019re unsure where to find it, execute the following: find / -name etcdctl Leveraging this tool to check the cluster status: etcdctl --write-out=table --endpoints=$ENDPOINTS endpoint status +------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | https://127.0.0.1:2379 | 4e30a295f2c3c1a4 | 3.5.0 | 8.1 MB | true | false | 3 | 7903 | 7903 | | +------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ The cluster this was executed on has only one master node, hence only one result from the script. You will normally receive a response for each etcd member in the cluster. Alternatively, leverage kubectl get componentstatuses: kubectl get componentstatuses #ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok etcd-1 Healthy {\"health\":\"true\"} etcd-0 Healthy {\"health\":\"true\"} Etcd may also be running as a Pod: kubectl logs etcd-ubuntu -n kube-system","title":"ETCD"},{"location":"revision-topics/05-troubleshooting/#kube-apiserver","text":"This is dependent on the environment for which the Kubernetes platform has been installed on. For systemd based systems: journalctl -u kube-apiserver Or cat /var/log/kube-apiserver.log Or for instances where Kube-API server is running as a static pod: kubectl logs kube-apiserver-k8s-master-03 -n kube-system","title":"Kube-apiserver"},{"location":"revision-topics/05-troubleshooting/#kube-scheduler","text":"For systemd-based systems journalctl -u kube-scheduler Or cat /var/log/kube-scheduler.log Or for instances where Kube-Scheduler is running as a static pod: kubectl logs kube-scheduler-k8s-master-03 -n kube-system","title":"Kube-Scheduler"},{"location":"revision-topics/05-troubleshooting/#kube-controller-manager","text":"For systemd-based systems journalctl -u kube-controller-manager Or cat /var/log/kube-controller-manager.log Or for instances where Kube-controller manager is running as a static pod: kubectl logs kube-controller-manager-k8s-master-03 -n kube-system","title":"Kube-Controller-Manager"},{"location":"revision-topics/05-troubleshooting/#worker-nodes","text":"","title":"Worker Node(s)"},{"location":"revision-topics/05-troubleshooting/#cni","text":"Obviously this is dependent on the CNI in use for the cluster you\u2019re working on. However, using Flannel as an example: journalctl -u flanneld If running as a pod, however: Kubectl logs --namespace kube-system <POD-ID> -c kube-flannel kubectl logs --namespace kube-system weave-net-pwjkj -c weave","title":"CNI"},{"location":"revision-topics/05-troubleshooting/#kube-proxy","text":"For systemd-based systems journalctl -u kube-proxy Or cat /var/log/kube-proxy.log Or for instances where Kube-proxy manager is running as a static pod: kubectl logs kube-proxy -n kube-system","title":"Kube-Proxy"},{"location":"revision-topics/05-troubleshooting/#kubelet","text":"journalctl -u kubelet Or cat /var/log/kubelet.log","title":"Kubelet"},{"location":"revision-topics/05-troubleshooting/#container-runtime","text":"Similarly to the CNI, this depends on which container runtime has been deployed, but using Docker as an example: For systemd-based systems: journalctl -u docker.service Or cat /var/log/docker.log Hint : list the contents of etc/systemd/system if it\u2019s a systemd-based service (containerd.service may be here)","title":"Container Runtime"},{"location":"revision-topics/05-troubleshooting/#cluster-logging","text":"At a cluster level, kubectl get events provides a good overview.","title":"Cluster Logging"},{"location":"revision-topics/05-troubleshooting/#understand-how-to-monitor-applications","text":"This section is a bit open-ended as it highly depends on what you have deployed and the topology of an application. Typically, however, we have an application that runs as a number of inter-connected microservices , consequently we monitor our applications by monitoring the underlying objects that comprise it, such as: Pods Deployments Services etc","title":"Understand how to monitor applications"},{"location":"revision-topics/05-troubleshooting/#manage-container-stdout-stderr-logs","text":"Kubernetes handles and redirects any output generated from a containers stdout and stderr streams. These get directed through a logging driver which influences where to store these logs. Different implementations of Docker differ in exact implementation (such as RHEL's flavor of Docker) but commonly, these drivers will write to a file in json format: root@ubuntu:~# docker info | grep \"Logging Driver\" Logging Driver: json-file The location for these logs is typically /var/log/containers but can be tweaked. Additionally, these contain symlinks: root@ubuntu:~# ls -la /var/log/containers/ total 44 drwxr-xr-x 2 root root 4096 Feb 8 19:18 . drwxrwxr-x 11 root syslog 4096 Feb 12 00:00 .. lrwxrwxrwx 1 root root 100 Feb 8 19:17 coredns-74ff55c5b-j4trd_kube-system_coredns-5d65324791ffcdf45d3552d875c6834f9a305c5be84b18745cb1657f784e5dd0.log -> /var/log/pods/kube-system_coredns-74ff55c5b-j4trd_4afbef57-5592-4edb-96af-9d17f595d160/coredns/0.log lrwxrwxrwx 1 root root 100 Feb 8 19:17 coredns-74ff55c5b-wrgkr_kube-system_coredns-b2fcfa679e9725dbe601bc1a0f218121a9c44b91d7300bbb57039a4edd219991.log -> /var/log/pods/kube-system_coredns-74ff55c5b-wrgkr_b64ac6d5-654b-4194-b6a8-5f8aa4c3cbe2/coredns/0.log lrwxrwxrwx 1 root root 81 Feb 8 19:16 etcd-ubuntu_kube-system_etcd-fcc5bc99932f380781776baa125b6f3be035e18fcec520afb827102e2afce1cd.log -> /var/log/pods/kube-system_etcd-ubuntu_f608198a8b73b3cf090bd15e2823df04/etcd/0.log lrwxrwxrwx 1 root root 101 Feb 8 19:16 kube-apiserver-ubuntu_kube-system_kube-apiserver-a264bbd54b7f23c8d424b0b368a48fdd1c5dcecc72fca95a460c146b2b5d85f5.log -> /var/log/pods/kube-system_kube-apiserver-ubuntu_212641053a16fa2bb404ccde20f6eaf0/kube-apiserver/0.log lrwxrwxrwx 1 root root 119 Feb 8 19:17 kube-controller-manager-ubuntu_kube-system_kube-controller-manager-a4ef7fe2b52272ea77f8de2da0989a9bcee757ae778fc08f1786b26b45bf13e1.log -> /var/log/pods/kube-system_kube-controller-manager-ubuntu_7bbe7d37f1b2c7586237165580c2f5c3/kube-controller-manager/0.log lrwxrwxrwx 1 root root 102 Feb 8 19:05 kube-flannel-ds-rfsfs_kube-system_install-cni-f8762e22fbf17925432682bdb1259a066208c62fa695d09cd6ee9b0cef3d36ba.log -> /var/log/pods/kube-system_kube-flannel-ds-rfsfs_2892d4e3-e326-4b4b-90c0-396fb80863ca/install-cni/0.log lrwxrwxrwx 1 root root 103 Feb 8 19:05 kube-flannel-ds-rfsfs_kube-system_kube-flannel-f96e019717814d7360e1aacd275cac121c13e0ee94cc5c93dcb35365608e6f83.log -> /var/log/pods/kube-system_kube-flannel-ds-rfsfs_2892d4e3-e326-4b4b-90c0-396fb80863ca/kube-flannel/0.log lrwxrwxrwx 1 root root 96 Feb 8 19:17 kube-proxy-l52f9_kube-system_kube-proxy-bfe08cb8663b46551e8608c094194ec61d03edfa7d25a6f414c07ed6563ada89.log -> /var/log/pods/kube-system_kube-proxy-l52f9_d2b73ed1-5df4-4a18-9595-20798db4f110/kube-proxy/0.log lrwxrwxrwx 1 root root 101 Feb 8 19:17 kube-scheduler-ubuntu_kube-system_kube-scheduler-4a695e53684f4591ec9385d6944f7841c0329aa49be220e5af6304da281cb41a.log -> /var/log/pods/kube-system_kube-scheduler-ubuntu_69cd289b4ed80ced4f95a59ff60fa102/kube-scheduler/0.log","title":"Manage container stdout &amp; stderr logs"},{"location":"revision-topics/05-troubleshooting/#troubleshoot-application-failure","text":"This is a somewhat ambitious topic to cover as how we approach troubleshooting application failures varies by the architecture of that application, which resources/API objects we're leveraging, if the application contains logs. However, good starting points would include running things like: kubectl describe <object> kubectl logs <podname> kubectl get events","title":"Troubleshoot application failure"},{"location":"revision-topics/05-troubleshooting/#troubleshoot-cluster-component-failure","text":"Covered in \"Evaluate cluster and node logging\"","title":"Troubleshoot cluster component failure"},{"location":"revision-topics/05-troubleshooting/#troubleshoot-networking","text":"","title":"Troubleshoot networking"},{"location":"revision-topics/05-troubleshooting/#dns-resolution","text":"Pods and Services will automatically have a DNS record registered against coredns in the cluster, aka \"A\" records for IPv4 and \"AAAA\" for IPv6. The format of which is: pod-ip-address.my-namespace.pod.cluster-domain.example my-svc-name.my-namespace.svc.cluster-domain.example Pod DNS records resolve to a single entity, even if the Pod contains multiple containers as they share the same networking space. Service DNS records resolve to the respective service object. Pods will automatically have their DNS resolution configured based on coredns settings. This can be validated by opening a shell to the pod and inspecting /etc/resolv.conf: > kubectl exec -it web-server sh kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead. / # cat /etc/resolv.conf nameserver 10.43.0.10 search default.svc.cluster.local svc.cluster.local cluster.local eu-central-1.compute.internal options ndots:5 10.43.0.10 being the coredns service object: > kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 16d To test resolution, we can run a pod with nslookup to test. For the pod below: > kubectl get po -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES web-server 1/1 Running 0 2d20h 10.42.1.31 ip-172-31-36-67 <none> <none> Knowing the format of the A record: pod-ip-address.my-namespace.pod.cluster-domain.example We should be able to resolve 10-42-1-31.default.pod.cluster.local . Tip : To determine the cluster domain, inspect the coredns configmap. Below indicating cluster.local . > kubectl get cm coredns -n kube-system -o yaml apiVersion: v1 data: Corefile: | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { Create a Pod with the tools required: kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml Test lookup: kubectl exec -i -t dnsutils -- nslookup 10-42-1-31.default.pod.cluster.local > kubectl exec -i -t dnsutils -- nslookup 10-42-1-31.default.pod.cluster.local Server: 10.43.0.10 Address: 10.43.0.10#53 Name: 10-42-1-31.default.pod.cluster.local Address: 10.42.1.31 Similarly, for a service, in this case a service called nginx-service that resides in the default namespace: > kubectl exec -i -t dnsutils -- nslookup nginx-service.default.svc.cluster.local Server: 10.43.0.10 Address: 10.43.0.10#53 Name: nginx-service.default.svc.cluster.local Address: 10.43.0.223 > kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx-service ClusterIP 10.43.0.223 <none> 80/TCP 9m15s","title":"DNS Resolution"},{"location":"revision-topics/05-troubleshooting/#cni-issues","text":"Mainly covered earlier in acquiring logs for the CNI. However, one issue that might occur is when a CNI is incorrectly, or not initialised. This may cause workloads to enter a pending status: kubectl get po -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx 0/1 Pending 0 57s <none> <none> <none> <none> kubectl describe <pod> can help identify issues with assigning IP addresses to nodes from the CNI","title":"CNI Issues"},{"location":"revision-topics/05-troubleshooting/#port-checking","text":"Similarly, with leveraging nslookup to validate DNS resolution in our cluster, we can lean on other tools to perform other diagnostic. All we need is a pod that has a utility like netcat , telnet etc.","title":"Port Checking"}]}
\ No newline at end of file