Talos singlenode: Errors when trying to update to 1.9.4 due to User "system:node:talos" cannot get resource "pods" in API group "" in the namespace "XXX"

[kastl-ars]

Hi all,

I am once again trying to upgrade my singlenode talos instance from 1.8.2 to anything newer. I tried 1.8.4 as well as 1.9.4. A while ago I tried both 1.8.4 and 1.9.1. Until now with now success, the drain step seems to always fail...

One thing I noticed are errors related to permissions of the system:node:talos user:

 user: warning: [2025-02-26T06:55:13.468086723Z]: [talos] WARNING: failed to evict pod: failed waiting on pod olm/olm-operator-699ffbd9b-gbgw9 to be deleted: 2 error(s) occurred:                                                                           
 user: warning: [2025-02-26T06:55:13.468088723Z]:  pod is still running on the node                                                                                                                                                                          
 user: warning: [2025-02-26T06:55:13.468091723Z]:  failed to get pod olm/olm-operator-699ffbd9b-gbgw9: pods "olm-operator-699ffbd9b-gbgw9" is forbidden: User "system:node:talos" cannot get resource "pods" in API group "" in the namespace "olm": no  relationship found between node 'talos' and this object

As this is a single node instance, it is logical that the pod cannot be evicted. But as I get a lot of errors of the same kind I was wondering what went wrong there...

Kind Regards,
Johannes

[kastl-ars]

Just for the record, the update to 1.9.4 went through this time, even with the above errors.

[smira]

This error looks like something that got already fixed in Talos 1.9.x, and it should never happen on 1.8.x, as it's specific to Kubernetes 1.32, which is only supported with 1.9.

[kastl-ars]

OK, I'll check during the next upgrades, now that I managed to update to 1.9.4 and k8s 1.32.x