unable to run "lxc cluster list" on cluster member Error: Failed to get the address of the cluster leader: context deadline exceeded

[smcavoy-workwork]

I am unable to run lxc cluster list on a cluster member. VMs continue to operate.

On dellserver7.internal.net

$ lxc cluster list
Error: Failed to get the address of the cluster leader: context deadline exceeded

on another node in the cluster

$ lxc cluster list -f compact
                NAME                            URL                   ROLES        ARCHITECTURE  FAILURE DOMAIN  DESCRIPTION  STATE        MESSAGE
  dellserver1.internal.net  https://22.33.44.101:8443                    x86_64        default                      ONLINE  Fully operational
  dellserver2.internal.net  https://22.33.44.103:8443  database          x86_64        default                      ONLINE  Fully operational
  dellserver3.internal.net  https://22.33.44.104:8443  database          x86_64        default                      ONLINE  Fully operational
  dellserver4.internal.net  https://22.33.44.102:8443  database-standby  x86_64        default                      ONLINE  Fully operational
  dellserver5.internal.net  https://22.33.44.105:8443  database-leader   x86_64        default                      ONLINE  Fully operational
                                                                 database
  dellserver6.internal.net  https://22.33.44.100:8443                     x86_64        default                      ONLINE  Fully operational
  dellserver7.internal.net  https://22.33.44.98:8443   database-standby    x86_64        default                      ONLINE  Fully operational
  dellserver8.internal.net  https://22.33.44.99:8443                      x86_64        default                      ONLINE  Fully operational

also on another node in the cluster:

~$ lxc cluster info dellserver7.internal.net
sysinfo:
  uptime: 1977908
  load_averages:
  - 6.15
  - 4.87
  - 5.05
  total_ram: 1081617436672
  free_ram: 41221533696
  shared_ram: 7471104
  buffered_ram: 1139605504
  total_swap: 0
  free_swap: 0
  processes: 1788
  logical_cpus: 72
storage_pools:
  default:
    space:
      used: 801812341248
      total: 1914481672192

log entries on dellserver7:

time="2025-02-20T10:02:32Z" level=error msg="Failed to get leader cluster member address" err="Failed to get the address of the cluster leader: context deadline exceeded"
time="2025-02-20T10:02:32Z" level=error msg="Failed to get leader cluster member address" err="Failed to get the address of the cluster leader: context deadline exceeded"
time="2025-02-20T10:38:05Z" level=warning msg="Failed to get database leader details" err="Failed to get the address of the cluster leader: context deadline exceeded"

[tomponline]

Please provide output of snap list on each cluster member.

[smcavoy-workwork]

Output of snap list on each member

dellserver1.internal.net
Name    Version      Rev    Tracking       Publisher    Notes
core22  20250110     1748   latest/stable  canonical**  base
lxd     6.2-bde4d03  31820  6/stable       canonical**  -
snapd   2.67         23545  latest/stable  canonical**  snapd

dellserver2.internal.net
Name    Version      Rev    Tracking       Publisher    Notes
core22  20250110     1748   latest/stable  canonical**  base
lxd     6.2-bde4d03  31820  6/stable       canonical**  -
snapd   2.67         23545  latest/stable  canonical**  snapd

dellserver3.internal.net
Name    Version      Rev    Tracking       Publisher    Notes
core22  20250110     1748   latest/stable  canonical**  base
lxd     6.2-bde4d03  31820  6/stable       canonical**  -
snapd   2.67         23545  latest/stable  canonical**  snapd

dellserver4.internal.net
Name    Version      Rev    Tracking       Publisher    Notes
core22  20250110     1748   latest/stable  canonical**  base
lxd     6.2-bde4d03  31820  6/stable       canonical**  -
snapd   2.67         23545  latest/stable  canonical**  snapd

dellserver5.internal.net
Name    Version      Rev    Tracking       Publisher    Notes
core22  20250110     1748   latest/stable  canonical**  base
lxd     6.2-bde4d03  31820  6/stable       canonical**  -
snapd   2.67         23545  latest/stable  canonical**  snapd

dellserver6.internal.net
Name    Version      Rev    Tracking       Publisher    Notes
core22  20250110     1748   latest/stable  canonical**  base
lxd     6.2-bde4d03  31820  6/stable       canonical**  -
snapd   2.67         23545  latest/stable  canonical**  snapd

dellserver7.internal.net
Name    Version      Rev    Tracking       Publisher    Notes
core22  20250110     1748   latest/stable  canonical**  base
lxd     6.2-bde4d03  31820  6/stable       canonical**  -
snapd   2.67         23545  latest/stable  canonical**  snapd

dellserver8.internal.net
Name    Version      Rev    Tracking       Publisher    Notes
core22  20250110     1748   latest/stable  canonical**  base
lxd     6.2-bde4d03  31820  6/stable       canonical**  -
snapd   2.67         23545  latest/stable  canonical**  snapd

[tomponline]

Could be high load or network latency between cluster members.

[tomponline]

Try doing systemctl reload snap.lxd.daemon on dellserver5.internal.net to move the database-leader role to another node and see if that helps.

[tomponline]

Or you could try stopping some of the workloads on dellserver7.internal.net and see if it improves.

[smcavoy-workwork]

Load is about 0-5 on across all systems (multi-core+socket)
IO wait time is 0
Network is possible but not immediately visible, everything is reachable. Will confirm

I will look at the reload snap.lxd.daemon this does not affect the running VMs there, correct?

[tomponline]

Correct.

[tomponline]

Could also try reloading LXD on dellserver7.internal.net too to ensure there's no issue with LXD generally.

[smcavoy-workwork]

I ran reload on dellserver7.internal.net and the issue is resolved,

log file rotated and new entries are:

time="2025-02-21T00:55:07Z" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
time="2025-02-21T00:55:08Z" level=warning msg="Dqlite: attempt 1: server 33.44.34.101:8443: no known leader"
time="2025-02-21T00:55:08Z" level=warning msg="Dqlite: attempt 1: server 33.44.34.102:8443: no known leader"
time="2025-02-21T00:55:08Z" level=warning msg="Dqlite: attempt 1: server 33.44.34.105:8443: context canceled"
time="2025-02-21T00:55:08Z" level=warning msg="Dqlite: attempt 1: server 33.44.34.98:8443: no known leader"
time="2025-02-21T00:55:08Z" level=warning msg="Dqlite: attempt 1: server 33.44.34.105:8443: context canceled"
time="2025-02-21T00:55:08Z" level=warning msg="Dqlite: attempt 1: server 33.44.34.100:8443: no known leader"
time="2025-02-21T00:55:08Z" level=warning msg="Dqlite: attempt 1: server 33.44.34.99:8443: no known leader"
time="2025-02-21T00:55:10Z" level=warning msg="Skipping attaching missing external interface" driver=bridge interface=ens2 network=lxdbr2 project=default

[smcavoy-workwork]

@tomponline the reload seems to have the unintended consequence of "disconnecting" running VMs from LXD management.
/snap/lxd/31820/bin/qemu-system-x86_64 continues to run VMs, along with virtiofsd but the VMs do not show as running within LXD (shown as stopped)

An attempt to start a VM where the virtiofsd or qemu-system-x86_64 is already running will result in an error

Error: Failed cleaning config drive mount path "/var/snap/lxd/common/lxd/devices/unbound-1/config.mount": Failed unmounting "/var/snap/lxd/common/lxd/devices/unbound-1/config.mount": Failed to unmount "/var/snap/lxd/common/lxd/devices/unbound-1/config.mount": device or resource busy

Is this expected or avoidable ?

[tomponline]

What does this command show:

sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- findmnt

[smcavoy-workwork]

here it is

~$ sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- findmnt
TARGET                                                              SOURCE                                   FSTYPE      OPTIONS
/                                                                   none                                     tmpfs       rw,relatime,inode64
├─/boot                                                             /dev/loop1[/boot]                        squashfs    ro,nodev,relatime,errors=continue
│ └─/boot                                                           /dev/sda2                                ext4        ro,relatime
├─/host                                                             /dev/loop1[/host]                        squashfs    ro,nodev,relatime,errors=continue
├─/meta                                                             /dev/loop1[/meta]                        squashfs    ro,nodev,relatime,errors=continue
├─/opt                                                              /dev/loop1[/opt]                         squashfs    ro,nodev,relatime,errors=continue
├─/srv                                                              /dev/loop1[/srv]                         squashfs    ro,nodev,relatime,errors=continue
├─/usr                                                              /dev/loop1[/usr]                         squashfs    ro,nodev,relatime,errors=continue
│ ├─/usr/share/ca-certificates                                      /dev/mapper/root_crypt[/usr/share/ca-certificates]
│ │                                                                                                          ext4        ro,relatime,discard
│ ├─/usr/share/misc                                                 tmpfs                                    tmpfs       rw,relatime,mode=755,inode64
│ ├─/usr/bin/kmod                                                   /dev/loop2[/wrappers/kmod]               squashfs    ro,relatime,errors=continue
│ ├─/usr/lib/modules                                                /dev/mapper/root_crypt[/usr/lib/modules] ext4        rw,relatime,discard
│ ├─/usr/lib/firmware                                               /dev/mapper/root_crypt[/usr/lib/firmware]
│ │                                                                                                          ext4        rw,relatime,discard
│ ├─/usr/src                                                        /dev/mapper/root_crypt[/usr/src]         ext4        rw,relatime,discard
│ └─/usr/lib/snapd                                                  /dev/loop0[/usr/lib/snapd]               squashfs    ro,nodev,relatime,errors=continue
├─/var                                                              /dev/loop1[/var]                         squashfs    ro,nodev,relatime,errors=continue
│ ├─/var/snap                                                       /dev/mapper/root_crypt[/var/snap]        ext4        rw,relatime,discard
│ │ ├─/var/snap/lxd/common/lxd/devlxd                               tmpfs                                    tmpfs       rw,relatime,size=100k,mode=755,inode64
│ │ ├─/var/snap/lxd/common/lxd/storage-pools                        /dev/mapper/root_crypt[/var/snap/lxd/common/lxd/storage-pools]
│ │ │                                                                                                        ext4        rw,relatime,discard
│ │ │ ├─/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/vm14
│ │ │ │                                                             lxd/virtual-machines/vm14                zfs         rw,relatime,xattr,posixacl
│ │ │ ├─/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/vm11
│ │ │ │                                                             lxd/virtual-machines/vm11             zfs         rw,relatime,xattr,posixacl
│ │ │ ├─/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/vm13
│ │ │ │                                                             lxd/virtual-machines/vm13           zfs         rw,relatime,xattr,posixacl
│ │ │ ├─/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/vm2
│ │ │ │                                                             lxd/virtual-machines/vm2             zfs         rw,relatime,xattr,posixacl
│ │ │ ├─/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/vm9
│ │ │ │                                                             lxd/virtual-machines/vm9          zfs         rw,relatime,xattr,posixacl
│ │ │ ├─/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/vm6
│ │ │ │                                                             lxd/virtual-machines/vm6          zfs         rw,relatime,xattr,posixacl
│ │ │ ├─/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/vm5
│ │ │ │                                                             lxd/virtual-machines/vm5       zfs         rw,relatime,xattr,posixacl
│ │ │ ├─/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/vm8
│ │ │ │                                                             lxd/virtual-machines/vm8          zfs         rw,relatime,xattr,posixacl
│ │ │ ├─/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/vm7
│ │ │ │                                                             lxd/virtual-machines/vm7          zfs         rw,relatime,xattr,posixacl
│ │ │ ├─/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/vm3
│ │ │ │                                                             lxd/virtual-machines/vm3          zfs         rw,relatime,xattr,posixacl
│ │ │ ├─/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/vm10
│ │ │ │                                                             lxd/virtual-machines/vm10           zfs         rw,relatime,xattr,posixacl
│ │ │ ├─/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/vm12
│ │ │ │                                                             lxd/virtual-machines/vm12           zfs         rw,relatime,xattr,posixacl
│ │ │ ├─/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/vm1
│ │ │ │                                                             lxd/virtual-machines/vm1              zfs         rw,relatime,xattr,posixacl
│ │ │ ├─/var/snap/lxd/common/lxd/storage-pools/default/virtual-machines/vm4
│ │ │ │                                                             lxd/virtual-machines/vm4          zfs         rw,relatime,xattr,posixacl
│ │ │ └─/var/snap/lxd/common/lxd/storage-pools/default/custom/default_images
│ │ │                                                               lxd/custom/default_images                zfs         rw,relatime,xattr,posixacl
│ │ ├─/var/snap/lxd/common/lxd/devices                              /dev/mapper/root_crypt[/var/snap/lxd/common/lxd/devices]
│ │ │                                                                                                        ext4        rw,relatime,discard
│ │ │ ├─/var/snap/lxd/common/lxd/devices/vm14/config.mount          lxd/virtual-machines/vm14[/config]       zfs         ro,relatime,xattr,posixacl
│ │ │ ├─/var/snap/lxd/common/lxd/devices/vm13/config.mount     lxd/virtual-machines/vm13[/config]  zfs         ro,relatime,xattr,posixacl
│ │ │ ├─/var/snap/lxd/common/lxd/devices/vm11/config.mount       lxd/virtual-machines/vm11[/config]    zfs         ro,relatime,xattr,posixacl
│ │ │ ├─/var/snap/lxd/common/lxd/devices/vm2/config.mount       lxd/virtual-machines/vm2[/config]    zfs         ro,relatime,xattr,posixacl
│ │ │ ├─/var/snap/lxd/common/lxd/devices/vm9/config.mount    lxd/virtual-machines/vm9[/config] zfs         ro,relatime,xattr,posixacl
│ │ │ ├─/var/snap/lxd/common/lxd/devices/vm6/config.mount    lxd/virtual-machines/vm6[/config] zfs         ro,relatime,xattr,posixacl
│ │ │ ├─/var/snap/lxd/common/lxd/devices/vm7/config.mount    lxd/virtual-machines/vm7[/config] zfs         ro,relatime,xattr,posixacl
│ │ │ ├─/var/snap/lxd/common/lxd/devices/vm3/config.mount    lxd/virtual-machines/vm3[/config] zfs         ro,relatime,xattr,posixacl
│ │ │ ├─/var/snap/lxd/common/lxd/devices/vm10/config.mount     lxd/virtual-machines/vm10[/config]  zfs         ro,relatime,xattr,posixacl
│ │ │ ├─/var/snap/lxd/common/lxd/devices/vm12/config.mount     lxd/virtual-machines/vm12[/config]  zfs         ro,relatime,xattr,posixacl
│ │ │ ├─/var/snap/lxd/common/lxd/devices/vm5/config.mount lxd/virtual-machines/vm5[/config]
│ │ │ │                                                                                                      zfs         ro,relatime,xattr,posixacl
│ │ │ ├─/var/snap/lxd/common/lxd/devices/vm8/config.mount    lxd/virtual-machines/vm8[/config] zfs         ro,relatime,xattr,posixacl
│ │ │ ├─/var/snap/lxd/common/lxd/devices/vm1/config.mount        lxd/virtual-machines/vm1[/config]     zfs         ro,relatime,xattr,posixacl
│ │ │ └─/var/snap/lxd/common/lxd/devices/vm4/config.mount    lxd/virtual-machines/vm4[/config] zfs         ro,relatime,xattr,posixacl
│ │ ├─/var/snap/lxd/common/ns                                       tmpfs                                    tmpfs       rw,relatime,size=1024k,mode=700,inode64
│ │ └─/var/snap/lxd/common/shmounts                                 tmpfs                                    tmpfs       rw,relatime,size=1024k,mode=711,inode64
│ │   ├─/var/snap/lxd/common/shmounts/lxcfs                         lxcfs                                    fuse.lxcfs  rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
│ │   └─/var/snap/lxd/common/shmounts/instances                     tmpfs                                    tmpfs       rw,relatime,size=100k,mode=711,inode64
│ ├─/var/lib/snapd                                                  /dev/mapper/root_crypt[/var/lib/snapd]   ext4        rw,relatime,discard
│ │ └─/var/lib/snapd/hostfs                                         /dev/mapper/root_crypt[/var/lib/snapd/hostfs]
│ │                                                                                                          ext4        rw,relatime,discard
│ │   └─/var/lib/snapd/hostfs                                       /dev/mapper/root_crypt                   ext4        rw,relatime,discard
│ │     ├─/var/lib/snapd/hostfs/dev                                 udev                                     devtmpfs    rw,nosuid,relatime,size=528092204k,nr_inodes=132023051,mode=755,inode6
│ │     ├─/var/lib/snapd/hostfs/proc                                proc                                     proc        rw,nosuid,nodev,noexec,relatime
│ │     ├─/var/lib/snapd/hostfs/boot                                /dev/sda2                                ext4        rw,relatime
│ │     │ └─/var/lib/snapd/hostfs/boot/efi                          /dev/sda1                                vfat        rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,sho
│ │     ├─/var/lib/snapd/hostfs/sys                                 sysfs                                    sysfs       rw,nosuid,nodev,noexec,relatime
│ │     │ └─/var/lib/snapd/hostfs/sys/firmware/efi/efivars          none                                     efivarfs    rw,relatime
│ │     ├─/var/lib/snapd/hostfs/run                                 tmpfs                                    tmpfs       rw,nosuid,nodev,noexec,relatime,size=105626704k,mode=755,inode64
│ │     │ ├─/var/lib/snapd/hostfs/run/user/1005                     tmpfs                                    tmpfs       rw,nosuid,nodev,relatime,size=105626700k,nr_inodes=26406675,mode=700,u
│ │     │ ├─/var/lib/snapd/hostfs/run/lock                          tmpfs                                    tmpfs       rw,nosuid,nodev,noexec,relatime,size=5120k,inode64
│ │     │ ├─/var/lib/snapd/hostfs/run/credentials/systemd-sysusers.service
│ │     │ │                                                         none                                     ramfs       ro,nosuid,nodev,noexec,relatime,mode=700
│ │     │ └─/var/lib/snapd/hostfs/run/snapd/ns                      tmpfs[/snapd/ns]                         tmpfs       rw,nosuid,nodev,noexec,relatime,size=105626704k,mode=755,inode64
│ │     ├─/var/lib/snapd/hostfs/images                              /dev/mapper/kvm_storage-images           ext4        rw,relatime
│ │     ├─/var/lib/snapd/hostfs/snap/snapd/23545                    /dev/loop0                               squashfs    ro,nodev,relatime,errors=continue
│ │     ├─/var/lib/snapd/hostfs/snap/core22/1748                    /dev/loop1                               squashfs    ro,nodev,relatime,errors=continue
│ │     ├─/var/lib/snapd/hostfs/snap/lxd/31820                      /dev/loop2                               squashfs    ro,nodev,relatime,errors=continue
│ │     └─/var/lib/snapd/hostfs/var/snap/lxd/common/ns              tmpfs                                    tmpfs       rw,relatime,size=1024k,mode=700,inode64
│ ├─/var/tmp                                                        /dev/mapper/root_crypt[/var/tmp]         ext4        rw,relatime,discard
│ └─/var/log                                                        /dev/mapper/root_crypt[/var/log]         ext4        rw,relatime,discard
├─/writable                                                         /dev/loop1[/writable]                    squashfs    ro,nodev,relatime,errors=continue
├─/dev                                                              udev                                     devtmpfs    rw,nosuid,relatime,size=528092204k,nr_inodes=132023051,mode=755,inode6
│ ├─/dev/ptmx                                                       devpts[/ptmx]                            devpts      rw,relatime,gid=5,mode=620,ptmxmode=666
│ ├─/dev/pts                                                        devpts                                   devpts      rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
│ │ └─/dev/pts                                                      devpts                                   devpts      rw,relatime,gid=5,mode=620,ptmxmode=666
│ ├─/dev/shm                                                        tmpfs                                    tmpfs       rw,nosuid,nodev,inode64
│ ├─/dev/hugepages                                                  hugetlbfs                                hugetlbfs   rw,relatime,pagesize=2M
│ └─/dev/mqueue                                                     mqueue                                   mqueue      rw,nosuid,nodev,noexec,relatime
├─/etc                                                              /dev/mapper/root_crypt[/etc]             ext4        rw,relatime,discard
│ ├─/etc                                                            tmpfs                                    tmpfs       rw,relatime,mode=755,inode64
│ ├─/etc/nsswitch.conf                                              /dev/loop1[/etc/nsswitch.conf]           squashfs    ro,nodev,relatime,errors=continue
│ ├─/etc/apparmor                                                   /dev/loop1[/etc/apparmor]                squashfs    ro,nodev,relatime,errors=continue
│ └─/etc/apparmor.d                                                 /dev/loop1[/etc/apparmor.d]              squashfs    ro,nodev,relatime,errors=continue
├─/home                                                             /dev/mapper/root_crypt[/home]            ext4        rw,relatime,discard
├─/root                                                             /dev/mapper/root_crypt[/root]            ext4        rw,relatime,discard
├─/proc                                                             proc                                     proc        rw,nosuid,nodev,noexec,relatime
│ └─/proc/sys/fs/binfmt_misc                                        systemd-1                                autofs      rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_i
│   └─/proc/sys/fs/binfmt_misc                                      binfmt_misc                              binfmt_misc rw,nosuid,nodev,noexec,relatime
├─/sys                                                              sysfs                                    sysfs       rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/security                                            securityfs                               securityfs  rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/cgroup                                                  cgroup2                                  cgroup2     rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/pstore                                                  pstore                                   pstore      rw,nosuid,nodev,noexec,relatime
│ ├─/sys/firmware/efi/efivars                                       efivarfs                                 efivarfs    rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/bpf                                                     bpf                                      bpf         rw,nosuid,nodev,noexec,relatime,mode=700
│ ├─/sys/kernel/debug                                               debugfs                                  debugfs     rw,nosuid,nodev,noexec,relatime
│ │ └─/sys/kernel/debug/tracing                                     tracefs                                  tracefs     rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/tracing                                             tracefs                                  tracefs     rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/fuse/connections                                        fusectl                                  fusectl     rw,nosuid,nodev,noexec,relatime
│ └─/sys/kernel/config                                              configfs                                 configfs    rw,nosuid,nodev,noexec,relatime
├─/tmp                                                              /dev/mapper/root_crypt[/tmp]             ext4        rw,relatime,discard
│ └─/tmp                                                            /dev/mapper/root_crypt[/tmp/snap-private-tmp/snap.lxd/tmp]
│                                                                                                            ext4        rw,relatime,discard
├─/run                                                              tmpfs                                    tmpfs       rw,nosuid,nodev,noexec,relatime,size=105626704k,mode=755,inode64
│ ├─/run/user/1005                                                  tmpfs                                    tmpfs       rw,nosuid,nodev,relatime,size=105626700k,nr_inodes=26406675,mode=700,u
│ ├─/run                                                            tmpfs                                    tmpfs       rw,nosuid,nodev,relatime,mode=755,inode64
│ ├─/run/lock                                                       tmpfs                                    tmpfs       rw,nosuid,nodev,noexec,relatime,size=5120k,inode64
│ ├─/run/credentials/systemd-sysusers.service                       none                                     ramfs       ro,nosuid,nodev,noexec,relatime,mode=700
│ ├─/run/snapd/ns                                                   tmpfs[/snapd/ns]                         tmpfs       rw,nosuid,nodev,noexec,relatime,size=105626704k,mode=755,inode64
│ └─/run/netns                                                      tmpfs[/netns]                            tmpfs       rw,nosuid,nodev,noexec,relatime,size=105626704k,mode=755,inode64
├─/media                                                            /dev/mapper/root_crypt[/media]           ext4        rw,relatime,discard
├─/mnt                                                              /dev/mapper/root_crypt[/mnt]             ext4        rw,relatime,discard
└─/snap                                                             /dev/mapper/root_crypt[/snap]            ext4        rw,relatime,discard
  ├─/snap/snapd/23545                                               /dev/loop0                               squashfs    ro,nodev,relatime,errors=continue
  ├─/snap/core22/1748                                               /dev/loop1                               squashfs    ro,nodev,relatime,errors=continue
  └─/snap/lxd/31820                                                 /dev/loop2                               squashfs    ro,nodev,relatime,errors=continue

[tomponline]

Try doing sudo nsenter --mount=/run/snapd/ns/lxd.mnt -- umount /var/snap/lxd/common/lxd/devices/vm1/config.mount

[smcavoy-workwork]

I get

umount: /var/snap/lxd/common/lxd/devices/vm1/config.mount: target is busy.

In the findmnt list I see VMs that are shown as stopped when lxc list but the qemu process is still running. I tried the above amount command on one those and received the same error, target is busy

[tomponline]

Looks like you still have a process running holding that mount open, take a look for virtfs-proxy-helper and virtiofsd.
You should also kill the relevant qemu process.