How to fix broken machine due to bad patch?

[justyns]

I've been testing various things with omni and a few times I've added a "bad" patch that causes nodes to get stuck in a reboot loop.

E.g., I tried adding this patch:

      machine:
        files:
          - content: ""
             op: overwrite
             path: /etc/cni/net.d/05-cilium.conf

It failed due to the file not existing, which is fine, but even after removing that part of the patch and saving it in omni - the affected nodes were still stuck in a reboot loop. They'd boot into Talos, have an error about that file not existing, and then reboot again. Rebooting the machine into maintenance mode seems to break it out of the loop and fix it, but that doesn't seem like the right way to do it.

Is there another way to fix nodes that are in a state like this?

Or is there a way to run talosctl apply-config using omnictl?

edit: For context, the log says this (different error for create vs overwrite, but otherwise the same), but I wasn't able to run either of these talosctl commands:

[talos] task writeUserFiles (1/1): writeUserFiles failed, rebooting in 35 minutes. You can use talosctl apply-config or talosctl edit mc to fix the issues, error:
create operation not allowed outside of /var: "/etc/cni/net.d/05-cilium.conf"

The node also reboots almost immediately - not 35 minutes later like it says.

[justyns]

I ran into a similar situation where I am adding disk.install to the machine config. It worked well for most servers, but I made a mistake on one and used /dev/sda when it should have been /dev/nvme0n1.

The machine boots up, eventually fails with an error saying the install disk doesn't exist and reboots. I fixed the machine config and did omnictl cluster template sync. Now omnictl cluster status has the node status listed as REBOOTING Config Outdated. After rebooting several times, it hasn't fixed itself. The error is still the same error about /dev/sda not existing.

Is there a way to force the node to get its updated config? It seems like it tries to apply the old config first.

[smira]

There are no logs to understand your issue, but if it is what you describe, than it shouldn't happen this way. If you update a config patch, it should be applied and the install happens as a result of applying machine config (or config validation?).

cc @Unix4ever

[Unix4ever]

Yeah, the failing node logs might be required here. Also you can check what is set in the machine config by doing: omnictl get redactedmachineconfig <node-id>.