Startup problems

[pwmarcz]

Hi, @marmarek's friend here, I'm trying to get the code to run and understand it so that we can bring the Qubes on KVM effort forward.

I am stuck trying to get anything to run. Could you give me some pointers? Unfortunately I'm not very familiar with qemu and libvirt.

(I'll be happy to fix these issues once I understand what's happening!)

• I cannot get it to work. I created a VM using virt-manager (x86_64 Ubuntu 20.04 host, and x86_64 Fedora 32 guest), and tried to run kvmchand. I get:

$ ./kvmchand 
[INFO] daemon/libvirt.c:530: Domain fedora(6, UUID: 63b6b932-f83a-48e7-9c0c-6eb91c908768) changed state (event: 4, detail: 0)!
[WARN] daemon/libvirt.c:604: Unknown lifecycle event 10! Ignoring...
[INFO] daemon/libvirt.c:530: Domain fedora(6, UUID: 63b6b932-f83a-48e7-9c0c-6eb91c908768) changed state (event: 2, detail: 0)!
[INFO] daemon/libvirt.c:262: About to attach ivshmem device at /tmp/kvmchand/ivshmem_socket, index 0
[INFO] daemon/ivshmem.c:534: Got connection from PID: 108108
[ERROR] daemon/libvirt.c:344: QEMU rejected ivshmem device: {"id":"libvirt-372","error":{"class":"GenericError","desc":"Bus 'pcie.0' does not support hotplugging"}}
[WARN] daemon/libvirt.c:552: Failed to attach ivshmem device! Ignoring...

I'm not sure how I should attach the device to work. I manually added "bus": "pci.1" to the QEMU request, and it seems to have done something, but kvmchand -g still fails with Unable to find ivshmem device.

• Restarting the daemon doesn't seem to work. When I kill the daemon with Ctrl-C, starting it again fails with:

[ERROR] daemon/libvirt.c:331: QEMU rejected chardev: {"id":"libvirt-396","error":{"class":"GenericError","desc":"attempt to add duplicate property 'charshmem0' to object (type 'container')"}}

It seems there is no graceful shutdown that would remove the devices from a VM? Or perhaps we should detect a situation where the devices have already been added.

• Restarting also fails with "Address already in use". Maybe the shutdown should delete the socket file? (And I would move the socket from /tmp to somewhere like /var/run).

[nrgaway]

I have it mostly working. I will update my repos within the next few hours. I will post the comments in #12.

[shawnanastasio]

To get up and running on x86_64 currently, the easiest way is to use an i440fx guest. Q35 guests will require some slight changes in both the guests's libvirt xml definition and libkvmchan's hotplug code. @nrgaway has been experimenting with this recently, and soon we should have Q35 guests working and documented.

In addition, you'll need to make sure the guest's kernel has CONFIG_VFIO_NOIOMMU set in order to use the guest-side daemon.

[nrgaway]

I pushed my changes to my repos. You can do a complete build for host and guest templates or just build
core-libvirt (kvm}
core-vchan-libkvmchan (master)
core-vchan-kvm (master)
core-qubesdb (kvm}
core-qrexec (kvm}

The following packages are required if you want to build all the Qubes packages for host and guest VM (template)
kvm (master)
linux-utils (kvm}
core-admin (kvm}
core-agent-linux (kvm}
linux-kernel (vfio_noiommu)

And a builder.conf file may be helpful.

All the packages build and template will start using qvm-start but there is still a long way to go before everything is fully functional. Some manual configuration of the host has to be preformed.

I also include a libvirt XML configuration that works with libkvmchan and Qubes, taking note of the machine type and PCI controller configuration.

<domain type='kvm'>
  <name>fedora-32</name>
  <uuid>12345678-1234-1234-1234-123456789012</uuid>
  <memory unit='KiB'>4096000</memory>
  <currentMemory unit='KiB'>4096000</currentMemory>
  <vcpu placement='static'>2</vcpu>

  <-- MACHINE, UEFI BIOS -->
  <os>
    <type arch='x86_64' machine='pc-i440fx-4.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/fedora-32_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>

  <features>
    <acpi/>
    <apic/>
    <vmport state='off'/>
  </features>
  <cpu mode='host-passthrough' check='none'/>
  <clock offset='variable' adjustment='0' basis='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>destroy</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>

    <-- DISK IMAGES -->
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/qubes/vm-templates/fedora-32/root-dirty.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/qubes/vm-templates/fedora-32/private-dirty.img'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/qubes/vm-templates/fedora-32/volatile-dirty.img'/>
      <target dev='vdc' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/qubes/vm-kernels/5.6.16-1/modules.img'/>
      <target dev='vdd' bus='virtio'/>
      <readonly/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/>
    </disk>

    <-- PCI CONTROLLERS -->
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='pci' index='1' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>

    <-- CONTROLLER USED FOR LIBVCHAN (with patches alreaded pre-appliedin core-vchan-libkvmchan] -->
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>

    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </controller>

    <-- HOST SHARES - REMOVE IF UNWANTED - ENABLED BY DEFAULT FOR DEVELOPMENT -->
    <filesystem type='mount' accessmode='squash'>
      <source dir='/home/kvm/share'/>
      <target dir='/share'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </filesystem>
    <filesystem type='mount' accessmode='squash'>
      <source dir='/home/kvm/qubes/qubes-packages-mirror-repo/vm-fc32/rpm'/>
      <target dir='/rpm'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </filesystem>

    <interface type='network'>
      <mac address='12:34:56:78:90:12'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>

   <-- CONSOLE ACCESS, no gui --->
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>

    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'>
      <listen type='address'/>
      <image compression='off'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0e' function='0x0'/>
    </memballoon>
    <rng model='virtio'>
      <backend model='random'>/dev/urandom</backend>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0f' function='0x0'/>
    </rng>
  </devices>
</domain>

Some manual VM configs may need to be performed depending on which Qubes packages you installed:

• Most important, be sure that a proper UEFI boot partition is configured. I have a WIP tool qvm-kvmraw-to-qubeskvm that will convert a raw KVM image created withvirt-manager that contains an efi, boot, swap and root partition to a Qubes partition layout and another that converts a Qubes template to properly contain an UEFI partition. The template that is built does not contain a configured UEFI partition yet.
• Run grub: grub2-mkconfig -o /etc/grub2-efi.cfg and grub2-mkconfig -o /etc/grub2.cfg.
• Make sure the kernel you have installed supports the VFIO_NOIOMMU option. The one Fedora ships does not. The kernel package I listed above includes support. You may need to update initramfs to include VFIO drivers.

$ sudo lsinitrd /boot/initramfs-$(uname -r).img | grep vfio
$ sudo dracut --force --add-drivers vfio-pci --kver $(uname -r)
# /etc/default/grub
GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX rd.driver.pre=vfio-pci"

• Check all systemd services to be sure none failed. You may need to disable (depending if feature enabled for the service) the following: qubes-iptables.service: core-agent-linux, qubes-network.service: core-agent-linux, qubes-gui-agent.service: gui-agent-linux, qubes-firewall.service: core-agent-linux, qubes-updates-proxy.service: core-agent-linux, qubes-updates-proxy-forwarder.socket: core-agent-linux
• Staring the VM:
• • If you build and install the Qubes host packages, start the VM using qubes-start. It will take care of creating the XML configuration automatically and connect to qubes-db and qrexec serivce.
    -- Otherwise, use virt-manager or as root, virsh start vmname --console. You may need to edit VM XML configuration disk location. You will need to start the host qubes-db and qrexec services manually.
• Disable Qubes repo in '/etc/yum.repos.d`

If you are brave enough to install the host components, only build for Fedora 32 Dom0 and VM and note I have only tested with BTRFS on the host since it's easier to debug and replace images when testing. Here is a list of manual configuration that is currently required.:

• touch /etc/qubes-release

• Add qubes user and group

• Add libvirt, kvm, qemu to qubes group (may not need them all)

• Make sure group is set to qubes for '/var/lib/qubes/*' group has 'rws' permissions

• Make sure qubes kernel is installed in '/var/lib/qubes/vm-kernels/5.6.16-1' (5.6.16-1 or whatever version you compiled)

• Disable Dom0 (host) dnf repo ('/etc/dnf/dnf.conf') by backing up the one installed by Qubes and replacing it with the original.

• Edit '/var/lib/qubes/qubes.xml' (maybe back it up first). Be sure the following config is in place before attempting to install qubes-template-fedora-32-4.0.1.noarch.rpm:

<labels>
    <label id="label-1" color="0xcc0000">red</label>
    <label id="label-2" color="0xf57900">orange</label>
    <label id="label-3" color="0xedd400">yellow</label>
    <label id="label-4" color="0x73d216">green</label>
    <label id="label-5" color="0x555753">gray</label>
    <label id="label-6" color="0x3465a4">blue</label>
    <label id="label-7" color="0x75507b">purple</label>
    <label id="label-8" color="0x000000">black</label>
  </labels>
  <pools>
    <pool name="varlibqubes" dir_path="/var/lib/qubes" driver="file-reflink" revisions_to_keep="1"/>
    <pool name="linux-kernel" dir_path="/var/lib/qubes/vm-kernels" driver="linux-kernel"/>
  </pools>
  <properties>
    <property name="clockvm"></property>
    <property name="default_dispvm"></property>
    <property name="default_kernel">5.6.16-1</property>
    <property name="default_netvm"></property>
    <property name="default_pool_kernel">linux-kernel</property>
    <property name="default_template">fedora-32</property>
    <property name="management_dispvm"></property>
    <property name="updatevm"></property>
  </properties>
  <domains>
    <domain id="domain-0" class="AdminVM">
      <properties>
        <property name="label">black</property>
      </properties>
      <features/>
      <devices class="usb"/>
      <tags/>
    </domain>
  <domains>

• Set up libvirt default networking to use same IP address range as Qubes to allow easier migration for when I specific networking has been implemented. sudo virsh net-edit default:

<network>
  <name>default</name>
  <!-- Keep same UUID? -->
  <uuid>12345678-1234-1234-1234-123456789012</uuid>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0'/>
  <! -- Keep same mac? -->
  <mac address='11:22:33:44:55:66'/>
  <ip address='10.137.0.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='10.137.0.2' end='10.137.0.254'/>
    </dhcp>
  </ip>
</network>

Hopefully some of this helps.

[nrgaway]

I also pushed post-installation steps after building Qubes-KVM which I'll keep updating as issues arise or resolved. I'm hoping to eliminate most, if not all, the VM post configuration tasks within the next week or so.

[pwmarcz]

Thanks! I was able to get the example for libkvmchan running with the help of your XML.

For now I'll be focusing on reading the base library, so I'm not installing Qubes yet, but the instructions will definitely come in handy later.