examples/cpem-add-on fails to provision

[displague]

The cpem-add-on example fails to provision:

cd examples/cpem-add-on
eval $(metal env -o terraform --export)
terraform init
terraform apply

This ultimately results in:

module.tfk8s.null_resource.kubeconfig (local-exec): ssh: connect to host 139.178.86.221 port 22: Connection timed out
╷
│ Error: local-exec provisioner error
│ 
│   with module.tfk8s.null_resource.kubeconfig,
│   on ../../main.tf line 119, in resource "null_resource" "kubeconfig":
│  119:   provisioner "local-exec" {
│ 
│ Error running command 'scp -o StrictHostKeyChecking=no -i ssh_key [email protected]:/etc/kubernetes/admin.conf kubeconfig.admin.yaml': exit status 1. Output: ssh: connect to host 139.178.86.221 port 22: Connection timed out
│ 
╵

This would appear to indicate that the control plane EIP was not assigned to a node in the cluster.
Upon logging in to the cp-1 node, we see a cloud-init provisioning failure in /var/log/cloud-init-output.log.

Setting up jq (1.6-1ubuntu0.20.04.1) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.14) ...
deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main
Get:1 https://download.docker.com/linux/ubuntu focal InRelease [57.7 kB]
Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease
Get:4 https://download.docker.com/linux/ubuntu focal/stable amd64 Packages [40.1 kB]
Hit:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease
Ign:3 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Err:6 https://packages.cloud.google.com/apt kubernetes-xenial Release
  404  Not Found [IP: 2607:f8b0:4023:1009::66 443]
Hit:7 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:8 http://security.ubuntu.com/ubuntu focal-security InRelease
Reading package lists...
E: The repository 'https://apt.kubernetes.io kubernetes-xenial Release' does not have a Release file.
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package kubelet
E: Unable to locate package kubeadm
E: Unable to locate package kubectl
E: Unable to locate package kubelet
E: Unable to locate package kubeadm
E: Unable to locate package kubectl
E: No packages found
/var/lib/cloud/instance/scripts/runcmd: 23: cannot create /etc/containerd/config.toml: Directory nonexistent
/var/lib/cloud/instance/scripts/runcmd: 23: containerd: not found
sed: can't read /etc/containerd/config.toml: No such file or directory
/var/lib/cloud/instance/scripts/runcmd: 25: kubeadm: not found
sed: can't read /etc/containerd/config.toml: No such file or directory
Failed to restart containerd.service: Unit containerd.service not found.
/var/lib/cloud/instance/scripts/runcmd: 32: ctr: not found
/var/lib/cloud/instance/scripts/runcmd: 33: cannot create /etc/kubernetes/manifests/vip.yaml: Directory nonexistent
/var/lib/cloud/instance/scripts/runcmd: 33: ctr: not found
Failed to enable unit: Unit file kubelet.service does not exist.
/var/lib/cloud/instance/scripts/runcmd: 42: kubeadm: not found
cp: cannot stat '/etc/kubernetes/admin.conf': No such file or directory
cp: cannot stat '/etc/kubernetes/admin.conf': No such file or directory
Cloud-init v. 23.4.3-0ubuntu0~20.04.1 running 'modules:final' at Fri, 22 Mar 2024 20:53:10 +0000. Up 39.41 seconds.
Cloud-init v. 23.4.3-0ubuntu0~20.04.1 finished at Fri, 22 Mar 2024 20:54:08 +0000. Datasource DataSourceEc2.  Up 97.33 seconds

[displague]

Using the new upstream repo gets past the apt update failure.

https://kubernetes.io/blog/2023/08/15/pkgs-k8s-io-introduction/#how-to-migrate

[displague]

After #63, provisioning continues to fail because the VIP EIP is used to source the SSH key:
scp -o StrictHostKeyChecking=no -i ssh_key root@{VIP_EIP_HERE}:/etc/kubernetes/admin.conf kubeconfig.admin.yaml

This could be sourced from cp-1 instead.