Connect QEMU workers to remote Talos master nodes

[kubebn]

I need your help and advice. I have a powerful GPU machine that I want to connect to the Talos AWS master node. I need to keep the machine running Ubuntu, but I want the Kubernetes (K8s) part to be managed by Talos. Since I only need one worker node (i.e., one VM), I don't want to deploy Proxmox or any hypervisor on the machine. It seems like using QEMU with GPU passthrough could be the right approach.

I understand that QEMU is mainly designed for development purposes, as outlined in this guide: https://www.talos.dev/v1.9/talos-guides/install/local-platforms/qemu/. However, I'm wondering if it can be configured to connect as just a worker node?

My knowledge of QEMU is limited, and I've already tried several methods to create a VM using plain QEMU, such as:

qemu-system-x86_64 \
-m 2048 \
-smp cpus=2 \
-cpu max \
-hda talos-nocloud.img \
-nographic \
-netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
-device virtio-net-pci,netdev=net0,mac=26:fd:a2:f3:80:7a \
-device virtio-rng-pci \
-device virtio-balloon,deflate-on-oom=on \
-monitor unix:/root/talos-default/talos-default-controlplane-1.monitor,server,nowait \
-no-reboot \
-boot order=cn,reboot-timeout=5000 \
-smbios type=1,uuid=3c1f20af-1436-4b3e-b42b-47e94ed4fd56 \
-chardev socket,path=/root/talos-default/talos-default-controlplane-1.sock,server=on,wait=off,id=qga0 \
-device virtio-serial \
-device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 \
-device i6300esb,id=watchdog0 \
-watchdog-action pause \
-drive id=virtio0,format=raw,if=none,file=/root/talos-default/talos-default-worker-1-0.disk,cache=none \
-device virtio-blk-pci,drive=virtio0,logical_block_size=512,physical_block_size=512 \
-machine q35,accel=kvm,smm=on \
-serial stdio \
-cdrom /root/iso/cidata.iso # trying nocloud tricks

&

qemu-system-x86_64 \
    -hda talos-nocloud.img \
    -cpu host \
    -netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
    -device virtio-net-pci,netdev=net0,mac=26:fd:a2:f3:80:7a \
    -net user \
    -machine type=q35,accel=kvm \
    -m 4000 \
    -nographic \
    -drive file=talos-vm.qcow2,if=virtio,format=qcow2 \
    -cdrom /root/iso/cidata.iso \
    -serial stdio \
    -monitor unix:/root/talos-default/talos-default-worker-1.monitor,server,nowait

in both cases I was getting

[    6.058905] [talos] volume status {"component": "controller-runtime", "controller": "block.VolumeManagerController", "volume": "EPHEMERAL", "phase": "waiting -> failed", "error": "no disks matched for volume"}

I wonder if anyone has encountered a similar case and can share some QEMU tricks that might work for this use case.

[smira]

Plain QEMU should work out of the box.

There is nothing specific to Talos in using QEMU, i.e. you can use any generic QEMU guides.

Either way, you can look into console logs to see more. The log itself you posted doesn't mean anything besides the fact that Talos can't allocate EPHEMERAL.

Most probably you're using a tiny disk image which simply doesn't have space to allocate a partition, and you should simply expand the original disk image to have bigger size to leave space for EPHEMERAL. Talos will handle the rest in terms of expanding things.

[kubebn]

Plain QEMU should work out of the box.

There is nothing specific to Talos in using QEMU, i.e. you can use any generic QEMU guides.

Either way, you can look into console logs to see more. The log itself you posted doesn't mean anything besides the fact that Talos can't allocate EPHEMERAL.

Most probably you're using a tiny disk image which simply doesn't have space to allocate a partition, and you should simply expand the original disk image to have bigger size to leave space for EPHEMERAL. Talos will handle the rest in terms of expanding things.

Thank you very much for your response. I actually got it running. However, I have two more Talos-related questions.

When using these flags with QEMU:

-kernel $KERNEL_FILE 
-initrd $INITRAMFS_FILE

I want to include NVIDIA GPU drivers and extensions. When I check the factory.image, I see that I have generated the following:

KERNEL_URL="https://factory.talos.dev/image/26124abcbd408be693df9fe852c80ef1e6cc178e34d7d7d8430a28d1130b4227/$TALOS_VERSION/kernel-amd64"
INITRAMFS_URL="https://factory.talos.dev/image/26124abcbd408be693df9fe852c80ef1e6cc178e34d7d7d8430a28d1130b4227/$TALOS_VERSION/initramfs-amd64.xz"

However, I noticed that the modules and extensions are still missing. After switching to:

install.image="factory.talos.dev/installer/26124abcbd408be693df9fe852c80ef1e6cc178e34d7d7d8430a28d1130b4227:v1.9.4"

it worked.

I have two questions:

Do I need a kernel and initramfs generated with NVIDIA drivers if the Talos is going to be installed through the image installer on the VM anyway?

Is there any way to use the Talos image cache with this QEMU setup (using -kernel and -initrd), or is it only possible when installing Talos via an ISO/disk image?

Thank you in advance!