Disk replacement, trying to understand where "Cannot import" is coming from

[Contik]

Hi!

On a virtual machine on a Proxmox hypervisor where the virtual machine has root-on-ZFS with ZFSBootMenu 2.3.0 and Arch Linux I'm trying to better understand a peculiarity that apparently gets triggered by disk replacement in a zpool.

After replacing the root/boot disk I see during the boot process:

cannot import 'zpool': no such pool or dataset
        Destroy and re-create the pool from
        a backup source.
cachefile import failed, retrying

Meanwhile everything seems to be working fine despite this message and a 2-second delay observed during boot. This is mainly me trying to see if I'm forgetting something and trying to understand what specifically is triggering this.

Here's a screen capture gif of how the VM usually boots as observed via the hypervisor's noVNC serial console emulator:

This to me is normal behavior:

1. ZFSBootMenu finds a dataset given in a bootfs pool property:

   # zpool get bootfs
   NAME   PROPERTY  VALUE                 SOURCE
   zpool  bootfs    zpool/root/archlinux  local
   

2. Since there's only one such dataset given in a bootfs pool property ZFSBootMenu accepts it as the default entry and after a 10-second timer boots into it

The VM has one disk with a vfat-formatted EFI partition 1 and a ZFS zpool partition 2:

# lsblk --paths --output 'NAME,SIZE,FSTYPE,PARTUUID,PARTTYPE,PARTTYPENAME,PTTYPE' /dev/sda
NAME         SIZE FSTYPE     PARTUUID                             PARTTYPE                             PARTTYPENAME PTTYPE
/dev/sda      38G                                                                                                   gpt
├─/dev/sda1  512M vfat       5d097af2-6288-4259-99e2-e2b403e10a8b c12a7328-f81f-11d2-ba4b-00a0c93ec93b EFI System   gpt
└─/dev/sda2 37.5G zfs_member c4566f34-721b-4754-98d8-a1b81b1281d4 6a85cf4d-1dd2-11b2-99a6-080020736631 Solaris root gpt

Partition UUID of /dev/sda1 (5d..8b) is used in EFI boot entries Boot0009 and Boot0008:

    # efibootmgr
    BootCurrent: 0009
    Timeout: 3 seconds
    BootOrder: 0009,0008,0002,0001,0003,0004,0005,0000,0007
    Boot0000* UiApp FvVol(7cb8bdc9-f8eb-4f34-aaea-3ee4af6516a1)/FvFile(462caa21-7614-4503-836e-8ab6f4662331)
    Boot0001* UEFI PXEv4 (MAC:BC241169A61D) PciRoot(0x0)/Pci(0x12,0x0)/MAC(bc241169a61d,1)/IPv4(0.0.0.0,0,DHCP,0.0.0.0,0.0.0.0,0.0.0.0){auto_created_boot_option}
    Boot0002* UEFI QEMU QEMU HARDDISK       PciRoot(0x0)/Pci(0x5,0x0)/Pci(0x1,0x0)/SCSI(0,0){auto_created_boot_option}
    Boot0003* UEFI PXEv6 (MAC:BC241169A61D) PciRoot(0x0)/Pci(0x12,0x0)/MAC(bc241169a61d,1)/IPv6([::],0,Static,[::],[::],64){auto_created_boot_option}
    Boot0004* UEFI HTTPv4 (MAC:BC241169A61D)        PciRoot(0x0)/Pci(0x12,0x0)/MAC(bc241169a61d,1)/IPv4(0.0.0.0,0,DHCP,0.0.0.0,0.0.0.0,0.0.0.0)/Uri(){auto_created_boot_option}
    Boot0005* UEFI HTTPv6 (MAC:BC241169A61D)        PciRoot(0x0)/Pci(0x12,0x0)/MAC(bc241169a61d,1)/IPv6([::],0,Static,[::],[::],64)/Uri(){auto_created_boot_option}
    Boot0007* EFI Internal Shell    FvVol(7cb8bdc9-f8eb-4f34-aaea-3ee4af6516a1)/FvFile(7c04a583-9e3e-4f1c-ad65-e05268d0b4d1)
--> Boot0008* ZFSBootMenu vmlinuz-backup    HD(1,GPT,5d097af2-6288-4259-99e2-e2b403e10a8b,0x800,0x100000)/\EFI\ZBM\vmlinuz-backup.EFI
--> Boot0009* ZFSBootMenu vmlinuz   HD(1,GPT,5d097af2-6288-4259-99e2-e2b403e10a8b,0x800,0x100000)/\EFI\ZBM\vmlinuz.EFI

And partition UUID of /dev/sda2 (c4..d4) is the machine's zpool drive:

# zpool status -P zpool
  pool: zpool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
config:

        NAME                                                          STATE     READ WRITE CKSUM
        zpool                                                         ONLINE       0     0     0
          /dev/disk/by-partuuid/c4566f34-721b-4754-98d8-a1b81b1281d4  ONLINE       0     0     0

I now attach a second disk to the machine that's only 10 GByte in size. This disk gets named /dev/sdb for now. I apply the same partitioning scheme I used for the first disk: partition 1 gets typecode EF00 (EFI System Partition), it begins at sector 2048, has a size of 512 MByte and gets vfat-formatted, partition 2 gets typecode BF00 (Solaris root partition), it takes up the remaining disk space. I then dd content from /dev/sda1 to /dev/sdb1:

dd if=/dev/sda1 of=/dev/sdb1 bs=1M

Since unused /dev/sdb2 is smaller than existing /dev/sda2 I can't zpool replace so instead I zpool add the new one and zpool remove the old one. After a few seconds data's evacuated from /dev/sda2 and the zpool now uses the new smaller partition with partition UUID d5..9b:

# zpool status -P zpool
  pool: zpool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
remove: Removal of vdev 0 copied 3.10G in 0h0m, completed on Sat Jan 25 18:50:25 2025
        38.3K memory used for removed device mappings
config:

        NAME                                                          STATE     READ WRITE CKSUM
        zpool                                                         ONLINE       0     0     0
          /dev/disk/by-partuuid/d568822e-db9c-462e-a49c-2e9ec7137b9b  ONLINE       0     0     0

errors: No known data errors

I then update EFI boot entries to use the partition UUIDs from what is at this point still called /dev/sdb1 (16..2a):

    # efibootmgr
    BootCurrent: 0009
    Timeout: 3 seconds
    BootOrder: 0008,0006,0002,0001,0003,0004,0005,0000,0007
    Boot0000* UiApp FvVol(7cb8bdc9-f8eb-4f34-aaea-3ee4af6516a1)/FvFile(462caa21-7614-4503-836e-8ab6f4662331)
    Boot0001* UEFI PXEv4 (MAC:BC241169A61D) PciRoot(0x0)/Pci(0x12,0x0)/MAC(bc241169a61d,1)/IPv4(0.0.0.0,0,DHCP,0.0.0.0,0.0.0.0,0.0.0.0){auto_created_boot_option}
    Boot0002* UEFI QEMU QEMU HARDDISK       PciRoot(0x0)/Pci(0x5,0x0)/Pci(0x1,0x0)/SCSI(0,0){auto_created_boot_option}
    Boot0003* UEFI PXEv6 (MAC:BC241169A61D) PciRoot(0x0)/Pci(0x12,0x0)/MAC(bc241169a61d,1)/IPv6([::],0,Static,[::],[::],64){auto_created_boot_option}
    Boot0004* UEFI HTTPv4 (MAC:BC241169A61D)        PciRoot(0x0)/Pci(0x12,0x0)/MAC(bc241169a61d,1)/IPv4(0.0.0.0,0,DHCP,0.0.0.0,0.0.0.0,0.0.0.0)/Uri(){auto_created_boot_option}
    Boot0005* UEFI HTTPv6 (MAC:BC241169A61D)        PciRoot(0x0)/Pci(0x12,0x0)/MAC(bc241169a61d,1)/IPv6([::],0,Static,[::],[::],64)/Uri(){auto_created_boot_option}
--> Boot0006* ZFSBootMenu vmlinuz-backup    HD(1,GPT,16397708-42e3-4a4d-b7e1-9209be07c02a,0x800,0x100000)/\EFI\ZBM\vmlinuz-backup.EFI
    Boot0007* EFI Internal Shell    FvVol(7cb8bdc9-f8eb-4f34-aaea-3ee4af6516a1)/FvFile(7c04a583-9e3e-4f1c-ad65-e05268d0b4d1)
--> Boot0008* ZFSBootMenu vmlinuz   HD(1,GPT,16397708-42e3-4a4d-b7e1-9209be07c02a,0x800,0x100000)/\EFI\ZBM\vmlinuz.EFI

I power off the VM, detach the old disk and start the VM. The machine still successfully boots but during boot I now see a 2-second pause:

After the lines:

running hook [zfs]
ZFS: Importing pool zpool

I now see

cannot import 'zpool': no such pool or dataset
        Destroy and re-create the pool from
        a backup source.
cachefile import failed, retrying

Since it says cachefile import failed I thought I might have to regenerate its cache file:

zpool set cachefile=/etc/zfs/zpool.cache zpool

This did not change ZFS' behavior. Similarly running generate-zbm doesn't change boot behavior. And honestly I'm stumbling in the dark with those commands as I do not fully grok what's causing this.

I'd gladly take any hints you might have. Where's Destroy and re-create the pool from a backup source. coming from, what's causing the 2-second delay and how do I treat this zpool correctly when replacing big-disk with small-disk? :)

Thanks!

[zdykstra]

Your distribution's initramfs is what imports the pool, it's what's generating the error message(s) you're seeing. Recreate your BE's initramfs and the errors will likely go away.

None of this, however, is related at all to ZFSBootMenu.

[zdykstra]

You also shouldn't be replacing the disk in a single disk pool like that. zfs send | zfs recv a recursive snapshot and call it good. Anything else is just asking for trouble.

[Contik]

Thank you for your pointers! Regenerating the machine's initial ramdisk did solve this behavior. Also now that I better understand its origin apologies for posting this as a ZFSBootMenu discussion.