Grub vs systemd for RAID1 on Hetzner bare metal servers?

[rochecompaan]

I'm working on some docs and scripts to streamline NixOS installation on Hetzner bare metal servers: https://github.com/rochecompaan/hetzner-nixos-deploy.

I have mixed results with systemd as a boot loader for RAID1 configs. Sometimes it works, and sometimes it doesn't. Even for servers where the BIOS claims to support UEFI, it doesn't always work. I have much better results with Grub and will likely make it the default in the Disko config generated, but I want to ensure I'm not missing something obvious. Here is an example of the generated systemd config:

{
  boot = {
    loader = {
      systemd-boot = {
        enable = true;
      };
      efi = {
        canTouchEfiVariables = true;
        efiSysMountPoint = "/boot";
      };
    };
  };

  disko.devices = {
    disk = {
      nvme0n1 = {
        type = "disk";
        device = "/dev/nvme0n1";
        content = {
          type = "gpt";
          partitions = {
            boot = {
              size = "1G";
              type = "EF00";
              content = {
                type = "filesystem";
                format = "vfat";
                mountpoint = "/boot";
              };
            };
            raid = {
              size = "100%";
              content = {
                type = "mdraid";
                name = "raid1";
              };
            };
          };
        };
      };
      nvme1n1 = {
        type = "disk";
        device = "/dev/nvme1n1";
        content = {
          type = "gpt";
          partitions = {
            raid = {
              size = "100%";
              content = {
                type = "mdraid";
                name = "raid1";
              };
            };
          };
        };
      };
    };
    mdadm = {
      raid1 = {
        type = "mdadm";
        level = 1;
        metadata = "1.2";
        content = {
          type = "gpt";
          partitions = {
            root = {
              size = "100%";
              content = {
                type = "filesystem";
                format = "ext4";
                mountpoint = "/";
              };
            };
          };
        };
      };
    };
  };
}

Here is the equivalent Grub config:

{
  boot = {
    loader = {
      grub = {
        enable = true;
        devices = [ "/dev/nvme0n1" "/dev/nvme1n1" ];
        efiSupport = true;
        efiInstallAsRemovable = true;
      };
      efi = {
        efiSysMountPoint = "/boot/efi";
      };
    };
  };

  disko.devices = {
    disk = {
      nvme0n1 = {
        type = "disk";
        device = "/dev/nvme0n1";
        content = {
          type = "gpt";
          partitions = {
            BOOT = {
              size = "1M";
              type = "EF02"; # for grub MBR
            };
            ESP = {
              size = "1G";
              type = "EF00";
              content = {
                type = "filesystem";
                format = "vfat";
                mountpoint = "/boot/efi";
              };
            };
            raid = {
              size = "100%";
              content = {
                type = "mdraid";
                name = "raid1";
              };
            };
          };
        };
      };
      nvme1n1 = {
        type = "disk";
        device = "/dev/nvme1n1";
        content = {
          type = "gpt";
          partitions = {
            BOOT = {
              size = "1M";
              type = "EF02"; # for grub MBR
            };
            ESP = {
              size = "1G";
              type = "EF00";
              content = {
                type = "filesystem";
                format = "vfat";
                mountpoint = "/boot-fallback/efi";
              };
            };
            raid = {
              size = "100%";
              content = {
                type = "mdraid";
                name = "raid1";
              };
            };
          };
        };
      };
    };
    mdadm = {
      raid1 = {
        type = "mdadm";
        level = 1;
        metadata = "1.2";
        content = {
          type = "gpt";
          partitions = {
            root = {
              size = "100%";
              content = {
                type = "filesystem";
                format = "ext4";
                mountpoint = "/";
              };
            };
          };
        };
      };
    };
  };
}

I haven't tried mirrored boots with Grub yet, hence the hacky /boot-fallback mount point for the second drive.

[sedlund]

maybe related:

https://github.com/nix-community/srvos/blob/1b225750753c6063744abe78e9977e7814475ea5/nixos/common/default.nix#L19-L22

also if you are not aware of srvos it has specific configs for hetzner hardware

[Mic92]

It's usually possible to enable uefi on all servers each days using the kvm console variant.
For Raid1, I would still suggest grub because we don't have mirrored boot for systemd-boot yet: https://github.com/Mic92/dotfiles/blob/3189d2db7e4d0ff8acf2a337cdd6f9929094e5ce/machines/eve/modules/disko.nix#L40
Grub can be installed in hybrid mode so it works with both UEFI/Bios.
This is useful because some new machines will always use UEFI and fail to boot otherwise.

[rochecompaan]

@sedlund I wasn't aware of srvos. Thanks for the pointer!

@Mic92 some of the servers we manage are from the server auction and might have an older BIOS. I didn't see an explicit option to enable UEFI on them, and the BIOS doesn't detect any bootable device when using systemd.

The comment you linked suggests that Grub doesn't boot the default entry. Either way, Grub is installed by nixos-anywhere, and I'm assuming it will install in a way that is compatible with the server.

Is mirroredBoots still necessary with Grub if one uses raid1 on the boot partition?

[Mic92]

@rochecompaan You would need to request the kvm console and enable it in the Firmware settings. EFI has been around for ages. I don't think you can still buy hardware from hetzner that doesn't have EFI support at all. It's just not the default.

I used mdraid before for the ESP partition, but I had some machines that wouldn't boot with this configuration. Mirrored boot however always works.

[rochecompaan]

@Mic92 that is exactly what I did, I ordered the KVM console and had it connected for 3 hours. A single option indicated that UEFI is enabled on a boot device, but the BIOS could not find a bootable device. I forgot what the exact configuration option was but I can check since the server in question isn't operational yet.

[Mic92]

Usually you don't need an default entry in the EFI. A proper EFI implementation will fallback to using disks with ESP partition. So far I only had one old work station that wouldn't boot otherwise.

[rochecompaan]

Grub with RAID1 and mirrored boots seem like the best option, then.

I tried out mirrored boots with the configuration below, but Grub complained that it couldn't find /boot, but it installed successfully on /boot0 and /boot1:

updating GRUB 2 menu...
installing the GRUB 2 boot loader into /boot0...
Installing for x86_64-efi platform.
Installation finished. No error reported.
updating GRUB 2 menu...
installing the GRUB 2 boot loader into /boot1...
Installing for x86_64-efi platform.
Installation finished. No error reported.
updating GRUB 2 menu...
installing the GRUB 2 boot loader on /dev/sda...
Installing for i386-pc platform.
Installation finished. No error reported.
installing the GRUB 2 boot loader on /dev/sdb...
Installing for i386-pc platform.
Installation finished. No error reported.
installing the GRUB 2 boot loader into /boot...
Installing for x86_64-efi platform.
/nix/store/df664cq4djbpzqb1xg4xaczqwc0w6zph-grub-2.12/sbin/grub-install: error: /boot doesn't look like an EFI partition.
/nix/store/grzd53670wn8fcqfn62slk468bx86ggc-install-grub.pl: installation of GRUB EFI into /boot failed: No such file or directory
Failed to install bootloader

Here is the config. I thought devices = [ "nodev" ] would prevent installation on /boot. Any ideas?

{
  boot = {
    loader = {
      grub = {
        enable = true;
        devices = [ "nodev" ];
        efiSupport = true;
        efiInstallAsRemovable = true;
        mirroredBoots = [ 
          { path = "/boot0"; devices = [ "nodev" ]; }
          { path = "/boot1"; devices = [ "nodev" ]; }
        ];
      };
    };
  };

  disko.devices = {
    disk = {
      sda = {
        type = "disk";
        device = "/dev/sda";
        content = {
          type = "gpt";
          partitions = {
            BOOT = {
              size = "1M";
              type = "EF02"; # for grub MBR
            };
            ESP = {
              size = "1G";
              type = "EF00";
              content = {
                type = "filesystem";
                format = "vfat";
                mountpoint = "/boot0";
              };
            };
            raid = {
              size = "100%";
              content = {
                type = "mdraid";
                name = "raid1";
              };
            };
          };
        };
      };
      sdb = {
        type = "disk";
        device = "/dev/sdb";
        content = {
          type = "gpt";
          partitions = {
            BOOT = {
              size = "1M";
              type = "EF02"; # for grub MBR
            };
            ESP = {
              size = "1G";
              type = "EF00";
              content = {
                type = "filesystem";
                format = "vfat";
                mountpoint = "/boot1";
              };
            };
            raid = {
              size = "100%";
              content = {
                type = "mdraid";
                name = "raid1";
              };
            };
          };
        };
      };
    };
    mdadm = {
      raid1 = {
        type = "mdadm";
        level = 1;
        metadata = "1.2";
        content = {
          type = "gpt";
          partitions = {
            root = {
              size = "100%";
              content = {
                type = "filesystem";
                format = "ext4";
                mountpoint = "/";
              };
            };
          };
        };
      };
    };
  };
}

[Mic92]

Can you check that nothing else set grub.devices somehow?

[rochecompaan]

I checked and don't have grub.devices defined anywhere else. The snippet I shared above is the only place where grub settings are defined in the config. I'm a bit stumped, but I will keep digging.

[hmajid2301]

Just to add here, I am experiencing the same issue. I copied the config you shared above, got the same error. I am 99% sure grub/boot settings are only getting set in with the disko config i.e. in my disk.nix file now.

As an aside its not directly related to this issue; I have 3 NVMEs and want to treat them as one disk but also boot for any of them if possible. i.e. use raid0 or lvm? Not sure (sorry don't want to hijack this issue, can create my own if needed). Also this was on a bare-metal mini PC.

Logs

updating GRUB 2 menu...
installing the GRUB 2 boot loader into /boot0...
Installing for x86_64-efi platform.
Installation finished. No error reported.
updating GRUB 2 menu...
installing the GRUB 2 boot loader into /boot1...
Installing for x86_64-efi platform.
Installation finished. No error reported.
updating GRUB 2 menu...
installing the GRUB 2 boot loader on /dev/nvme0n1...
Installing for i386-pc platform.
Installation finished. No error reported.
installing the GRUB 2 boot loader on /dev/nvme1n1...
Installing for i386-pc platform.
Installation finished. No error reported.
installing the GRUB 2 boot loader into /boot...
Installing for x86_64-efi platform.
/nix/store/m55w361vnm6w18r0q6pf437w2yjzyipi-grub-2.12/sbin/grub-install: error: /boot doesn't look like an EFI partition.
/nix/store/gdf8k47sxkhfq56w5g82qr5g7ygmdqli-install-grub.pl: installation of GRUB EFI into /boot failed: No such file or directory
Failed to install bootloader

Link to my config: https://gitlab.com/hmajid2301/nixicle/-/blob/c1f6da2184b01b9c760626e6a9689c7b361aa1ab/systems/x86_64-linux/ms01/disks.nix

Thanks!