Extra NVMe storage on bare metal worker

[cmusser]

I have a bare-metal Talos cluster (3 control plane, 1 worker, Talo 1.9) and I wanted to have storage for persistent volumes. I installed an extra NVMe device in the worker node and need help setting it up.

I looked through the admin guides and in the generated machine config and have some idea how to do it. In broad strokes:

1. Use talos wipe disk, to remove the current data. In my case the disk had partitions for a Windows install, which aren't needed obviously.

2. Alter the disks section of the machine configuration to mention the new disk. There's a partitions subsection where you specify a mountpoint and a size. I guess the system takes care of doing the partitioning and formatting?

3. Alter the kubelet section's extraMounts to have the new mountpoint. I'm less sure that this is necessary. Also , the generated machine config' example section had stuff I didn't understand: mount options bind and rshared.

4. Reboot to have it take effect.

Step 2 and 3 would be done by talosctl patch mc. Not sure if "strategic merge patches" or JSON patches are more appropriate. I am adding new things in both cases.

Do those steps look more or less correct, or are there other concepts at play?

[smira]

Alter the disks section of the machine configuration to mention the new disk. There's a partitions subsection where you specify a mountpoint and a size. I guess the system takes care of doing the partitioning and formatting?

Yes, it will, and that's the way right now. Talos 1.10 should provide a more flexible way. Changing .machine.disks will trigger a reboot and formatting.

Alter the kubelet section's extraMounts to have the new mountpoint. I'm less sure that this is necessary. Also , the generated machine config' example section had stuff I didn't understand: mount options bind and rshared.

bind mount propagates bound from the host down to the kubeler, rshared is mount propagation, keep it this way.

Reboot to have it take effect.

No need to reboot (or Talos will reboot automatically).

Step 2 and 3 would be done by talosctl patch mc. Not sure if "strategic merge patches" or JSON patches are more appropriate. I am adding new things in both cases.

Strategic merge patches are easier, so use them.

Do those steps look more or less correct, or are there other concepts at play?

Yes, if you want just "raw" storage.

You can use extra NVMe disks for Ceph/OpenEBS, this way you don't need to partition and mount them directly, and storage subsystem will take care of managing it, replicating, etc.

[cmusser]

This worked for me. Thank you @smira for the assist. Having used the system for a while, I am going to take your advice about using those NVMe devices with an actual storage system. I'm going to try Mayastor.