NSX Components in K8S Integration
Ubuntu 18.04 Installation
Docker Installation
K8S Installation
This section focuses only on the differences between NSX-T 2.4 and NSX-T 2.5' s integration with K8S. For the detailed explanation of how the architecture looks like please refer to Part 2 of the prevous series (K8S with NSX-T 2.4.x).
-
First difference with NCP 2.5 is that previously the admin had to login to each K8S node; then install and configure CNI Plugin and Open vSwitch (OVS) on each node seperately and also upgrade them manually. Now this is all managed and maintained by a new K8S daemonset called "NSX-NCP-Bootstrap". "NSX-NCP-Bootstrap" Pod on each K8S node bootstraps the CNI Plugin and OVS on each node and attaches the selected ethernet interface on the respective node to the OVS. "NSX-NCP-Bootstracp" Pod has a single container in it which is named as "nsx-dummy".
-
Second difference is a new/additional container called "NSX-OVS" which is packaged within the well known "NSX Node Agent" Pod. "NSX-OVS" container keeps the OVS daemon running on the K8S node. Which means the OVS daemon is run in the container' s IPC namespace. When someone logins to the K8S node itself he/she will not be able to perform OVS commands on the host itself.
Diagram highlighting the recent changes mentioned above with NCP 2.5 is shown below.
-
Third difference is the manifest for all the NSX-T related components are now packaged in a single YAML file. Those are, NSX Container Plugin (NCP) as a deployment, plus all the config parameters with it (NCP.ini as configmap ), the NSX infrastructure' s own namespace "nsx-system" , the role based access control architecture that is applied to that namespace, NSX NCP Bootstrap daemonset and NSX Node Agent as daemonset.
-
Fourth difference is the NSX-T Policy API support with K8S. As mentioned in Part 1, Policy API corresponds to Simplified UI in the NSX-T GUI. Usage of Policy API is presented as an option in the NCP config.
-
Fifth difference is a new alternative topology option that is introduced. In all previous releases of NCP and NSX, each K8S namespace in a given K8S cluster used to have its own Tier1 Gateway/Logical Router at all times. Source NAT for each K8S Namespace was configured on Tier 0 Gateway/Logical Router and also K8S Ingress and Service Type Load Balancer for each K8S cluster was implemented on a seperate/dedicated Tier 1 Gateway. With NSX-T 2.5, there is an option ("single_tier_topology") which is presented in NCP config. When that option is set to "True" then all these services are collapsed on a single Tier 1 Gateway. When a new namespace is provisioned that will be realized as a new downlink segment attached to that single Tier 1 Gateway. All K8S Pods in the cluster also get Source NATed on that single Tier 1 Gateway. K8S Ingress and Service Type Load Balancer VIPs are also realized on that single Tier 1 Gateway. In this kind of a topology since stateful services are configured on Tier 1 layer, Tier 0 Gateway can be run in active active mode which provides higher north south throughput.
Note : If desired the topology can be simplified further as following; the Tier 1 Gateway/Logical Router, the one which is already configured in this demonstration as "T1-K8S-NodeManagement", to which the K8S Nodes' management ethernet is connected, can be used as the single/collapsed Tier 1 Gateway/Logical Router for all functions mentioned above. The related parameter to be used in NCP config is "top_tier_router" . When this parameter is set to the existing "K8S-NodeManagement" Tier 1 then that Tier 1 Gateway/logical router will be the single/collapsed Tier 1 router to be used for the K8S cluster (for K8S Node Management, K8S namespace/POD connectivity with or without NAT, K8S services (Load Balancing), K8S Ingress (Layer 7 Load Balancing).
Below diagram shows the new optional topology that can be used with the introduction of NCP 2.5. The IP address spaces mentioned in this diagram will be explained in the next post.
As mentioned just above, this topology can be further simplified by using a collapsed Tier 1 by configuring the respective NCP config parameter ("top-tier-router") with an existing Tier 1 Router' s name/UUID in the NSX-T domain. In this lab these Tier 1 Gateway/Logical Router' s kept seperate, meaning "top-tier-router" parameter is left blank hence NSX-T creates a new Tier 1 Gateway/Logical Router for the K8S cluster. In the above diagram Tier 1 with Load Balancer runs in active/standby mode (meaning that it has both SR and DR component) however the other Tier1 for K8S Node Management connectivity only runs the DR component since stateful services (NAT, LB etc.) is NOT used on that Tier 1.
-
Sixth difference is the K8S Network Policy driven NSX-T DFW rules can be placed in the "Application" category of the DFW in the simplified UI. (with three different options as allow cluster, allow namespace, none) This is also presented as a new parameter in NSX Config (as "baseline_policy_type")
-
There are additional new parameters introduced in NCP config which will be explained in the upcoming parts.
For this lab three VM' s will be configured. Below is a screenshot from one of the VM' s hardware configuration for Ubuntu 18.04 OS 64 bit.
Note#1 : As mentioned in Part 1; two network interfaces will be used on each Ubuntu node.
Note2 : As mentioned in NCP 2.5 Release Notes, NCP is compatible with Linux Kernel version lower than 4.15.0-59. Hence it is highly recommended to make sure not to update your Linux Kernel version when performing "apt-get update" . To achieve that, the steps mentioned in this article should be followed.
For Ubuntu 18.04 installation please refer to this article which outlines the steps needed in a simplified flow structure.
Additional steps needed to be followed to prepare the Ubuntu OS for Docker and K8S installation. Repeat Steps 1-5 below for all three nodes. In this lab "k8s-master" , "k8s-worker1" and "k8s-worker2" are configured. (Either use sudo or escalate to shell for root privileges)
- Configure a unique hostname :
sudo hostnamectl set-hostname k8s-master
- Configure the hosts file
sudo vi /etc/hosts
127.0.1.1 k8s-master
Not all nodenames are included in the hosts file of each node, as DNS is used in this lab. If DNS does not exist then since each node should be able to access each other by name, all nodenames and IP addresses must be included in the hosts file on each node.
- Check the interfaces on the node
vmware@k8s-master:~$ ip link show
1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens160: mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 00:50:56:b4:28:b8 brd ff:ff:ff:ff:ff:ff
3: ens192: mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT group default qlen 1000
link/ether 00:50:56:b4:0a:52 brd ff:ff:ff:ff:ff:ff
- Configure a static IP on the first interface
sudo vi /etc/netplan/50-cloud-init.yaml
Sample Config :
network:
version: 2
renderer: networkd
ethernets:
ens160:
dhcp4: no
addresses:
- 10.190.22.10/24
gateway4: 10.190.22.1
nameservers:
search: [demo.local]
addresses: [192.168.1.185]
Apply Changes :
sudo netplan apply
Verify Changes :
vmware@k8s-master:~$ ip address show dev ens160
2: ens160: mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:b4:28:b8 brd ff:ff:ff:ff:ff:ff
inet 10.190.22.10/24 brd 10.190.22.255 scope global ens160
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:feb4:28b8/64 scope link
valid_lft forever preferred_lft forever
vmware@k8s-master:~$
There should not be any IPv4 addresses configured on ens192 interface, which can be verified as below.
vmware@k8s-master:~$ ip address show dev ens192
3: ens192: mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
link/ether 00:50:56:b4:0a:52 brd ff:ff:ff:ff:ff:ff
inet6 fe80::250:56ff:feb4:a52/64 scope link
valid_lft forever preferred_lft forever
vmware@k8s-master:~$
- Turn off Swap on each node
sudo swapoff -a
Comment out the line with swap in the "fstab" configuration
sudo vi /etc/fstab
Sample Outputs is shown below :
vmware@k8s-master:~$ sudo cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
#
# / was on /dev/sda2 during curtin installation
/dev/disk/by-uuid/ca480e8b-f76f-4429-acc1-50e871ec925e / ext4 defaults 0 0
/swap.img none swap sw 0 0
Comment out the last line in the above output where it says "/swap.img" and save it.
Why turn off swap ? Additional Information is here :
K8S requires the SWAP to be disabled => https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.8.md#before-upgrading
There is a great blog article about why this step is needed => https://frankdenneman.nl/2018/11/15/kubernetes-swap-and-the-vmware-balloon-driver/
How to do it on Ubuntu => https://www.tecmint.com/disable-swap-partition-in-centos-ubuntu/
- Install Python
On each node install Python by running "apt-get install python"
Note : Python is needed< on each node for CNI Plugin to successfully send calls to NSX Node Agent.
To be compliant with the versions listed in NCP 2.5 Release Notes certain version of K8S will be installed. For this lab K8S 1.14 will be installed. To check the compatibility of K8S <-> Docker this URL can be reviewed. For this lab Docker 18.06 will be installed.
Install Docker on all nodes.
- Escalate to root in the shell (if not already)
vmware@k8s-master:~$ sudo -H bash
[sudo] password for vmware:
root@k8s-master:/home/vmware#
OR use "sudo" with each command shown below
Ensure the integrity and authenticity of the images that are downloaded from Docker Hub. GPG is based on Public Key Cryptogragphy (more info here : https://www.gnupg.org/)
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
Configure Docker Hub as the APT source rather than the Ubuntu 18.04 repository
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get update
vmware@k8s-master:~$sudo apt-cache policy docker-ce
docker-ce:
Installed: 18.06.0~ce~3-0~ubuntu
Candidate: 5:19.03.5~3-0~ubuntu-bionic
Version table:
5:19.03.5~3-0~ubuntu-bionic 500
500 https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
5:19.03.4~3-0~ubuntu-bionic 500
------ OUTPUT OMITTED ------
------ OUTPUT OMITTED ------
------ OUTPUT OMITTED ------
18.06.0~ce~3-0~ubuntu 500
500 https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
100 /var/lib/dpkg/status
18.03.1~ce~3-0~ubuntu 500
500 https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
vmware@k8s-master:~$
sudo apt-get install docker-ce=18.06.0~ce~3-0~ubuntu
Note : To make sure of an "apt-get update" not to break the compatibility between Docker and K8S, it would be a good practice to apply apt-mark hold on the related components. For example by using "apt-mark hold docker-ce".
- Verify Docker version
vmware@k8s-master:~$ sudo docker version
[sudo] password for vmware:
Client:
Version: 18.06.0-ce
API version: 1.38
Go version: go1.10.3
Git commit: 0ffa825
Built: Wed Jul 18 19:09:54 2018
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 18.06.0-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: 0ffa825
Built: Wed Jul 18 19:07:56 2018
OS/Arch: linux/amd64
Experimental: false
vmware@k8s-master:~$
- Verify Docker is running (if not, "sudo systemctl enable docker")
vmware@k8s-master:~$ sudo systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2019-12-03 21:13:40 UTC; 1 weeks 2 days ago
Docs: https://docs.docker.com
Main PID: 1013 (dockerd)
Tasks: 290
CGroup: /system.slice/docker.service
├─ 1013 /usr/bin/dockerd -H fd://
├─ 1200 docker-containerd --config /var/run/docker/containerd/containerd.toml
├─ 2762 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/6b70
├─ 2766 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/b041
├─ 2767 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/be92
├─ 2779 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/031a
├─ 2839 /pause
├─ 2874 /pause
├─ 2875 /pause
├─ 2876 /pause
├─ 2987 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/fb6f
├─ 3012 kube-scheduler --bind-address=127.0.0.1 --kubeconfig=/etc/kubernetes/scheduler.conf --leader-elect=true
├─ 3028 docker-cont
------ OUTPUT OMITTED ------
------ OUTPUT OMITTED ------
------ OUTPUT OMITTED ------
Follow the three steps below (until "Install Kubernetes" section) on all nodes.
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
vmware@k8s-master:~$ sudo apt-cache policy kubelet
kubelet:
Installed: 1.14.7-00
Candidate: 1.17.0-00
Version table:
1.17.0-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.16.4-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.16.3-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.16.2-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
------ OUTPUT OMITTED ------
------ OUTPUT OMITTED ------
------ OUTPUT OMITTED ------
1.14.8-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
1.14.7-00 500
500 http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
100 /var/lib/dpkg/status
------ OUTPUT OMITTED ------
------ OUTPUT OMITTED ------
------ OUTPUT OMITTED ------
vmware@k8s-master:~$
Same check can be done for "kubectl" and "kubeadm"
- Escalate to root in the shell (if not already)
vmware@k8s-master:~$ sudo -H bash
[sudo] password for vmware:
root@k8s-master:/home/vmware#
OR use "sudo" with each command shown below
sudo apt-get install -y kubectl=1.14.7-00 kubeadm=1.14.7-00 kubelet=1.14.7-00
sudo kubeadm init
At the end of the installation process an output similar too the below one should be observed.
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 10.190.22.10:6443 --token txy6bc.o9lpnabsxin9vavyfa \
--discovery-token-ca-cert-hash sha256:9824b49e691c3ab0dcdeb97b26a57bbb7abf619393a332786a336f2ff5b560e0
- Check the cluster state
At this stage kubelet will provide the following error, which can be seen in the output of "kubectl describe nodes" output.
Ready False Tue, 03 Dec 2019 21:27:26 +0000 Tue, 03 Dec 2019 21:19:26 +0000 KubeletNotReady runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
The node will show up in "NotReady" state since the CNI plugin is not installed on any node yet. This is expected.
vmware@k8s-master:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master NotReady master 10m v1.14.7
vmware@k8s-master:~$
sudo apt-get install -y kubectl=1.14.7-00 kubeadm=1.14.7-00 kubelet=1.14.7-00
Note : Kubectl would not be needed on worker nodes but there is no harm in installing it on worker nodes
Use the command mentioned below on "k8s-worker1" and "k8s-worker2" to join them to the K8S cluster.
vmware@k8s-worker1:~$ sudo kubeadm join 10.190.22.10:6443 --token txy6bc.o9lpnabsxin9vavyfa \
--discovery-token-ca-cert-hash sha256:9824b49e691c3ab0dcdeb97b26a57bbb7abf619393a332786a336f2ff5b560e0
Note : "kubectl get nodes" output will still show all nodes in "NotReady" state. This is expected.
Note 2 : "kubectl get pods -o wide --all-namespaces" will show "core-dns-xxx" Pods status stuck in "ContainerCreating" state. This is because of the same reason and that is CNI Plugin is not ready yet on the K8S nodes.