Upgrade UToronto cluster's kubernetes version

[yuvipanda]

(Refined and owned by @sgibson91)

Context

Needs #4669 to be completed.

UToronto's cluster is still on Kubernetes v1.28, which reaches EOL at the end of October. We should upgrade it. However, we currently do not have any documentation for upgrading Azure clusters.

Tasklist

Tasks

Preview Give feedback

1. Use learnings of [Spike: 2hr] Learn how to upgrade k8s version of clusters running on Azure #4669 to upgrade the control plane and nodepools of the UToronto cluster
2. Coordinate with community champion (as defined in relevant airtable) via freshdesk to establish an acceptable maintenance window for potential downtime Using rolling upgrades to not require this where possible
3. Add documentation to our infrastructure repo on how to upgrade the k8s version on Azure clusters

Definition of Done

• UToronto cluster's control plane and nodepools are running the most recent stable version of k8s Azure currently offers
• Documentation on upgrading k8s versions on Azure clusters is added

Pre-defined Definition of Done

• All the tasks have been completed
• It has been tested to work and to not break any existing functionality
• Documentation was added so other engineers can implement this in the future if needed

[yuvipanda]

@sgibson91 can you:

1. Refine this issue
2. Tell me how your experience refining it was?

we can discuss the experience during our next 1:1! If you get blocked midway, that's fine too. I'm learning how to best support others in refining tasks.

[sgibson91]

@yuvipanda I gave this a shot too, let's sync about it

[yuvipanda]

Thanks @sgibson91. This looks good to me. Let's do the spike in #4669 first, and then after the outcome of that, determine a time for scheduling the rest of the work.

[yuvipanda]

Me and @sgibson91 have decided to defer this to october, although #4669 will be done before.

[sgibson91]

#4669 has been completed so this is no longer blocked

[sgibson91]

As of #4851, the control plane and core node pool are running k8s v1.30.3 and there is a new user node pool running the same. Waiting for old pool to drain before I delete it.

Running notes of the experience are here (to be turned into docs): https://hackmd.io/@sgibson91/BJ0hFkoT0

[sgibson91]

Removed the tainted node, I just need to write up the docs now.