Partition tables

[achilleas-k]

Following up from the partition table and filesystem customization discussion in osbuild/bootc-image-builder#18.

---

Some relevant quotes from the discussion:
@cgwalters osbuild/bootc-image-builder#18 (comment)

... I think the default filesystem type should come from the container image. We could add bootc install --introspect which would output a merged JSON file from all the bootc install configuration files that live in the image.

@achilleas-k osbuild/bootc-image-builder#18 (comment)

I think it makes sense to have an agreed-upon default partition table at each level of the image building process. That is, if a container doesn't specify a PT, BIB should still be able to build a disk image, or generate a kickstart, with the default partition table for the distro.
... if bootc will be able to produce a partition table for BIB to consume in order to generate the osbuild manifest, the representation of that partition table will effectively be an "input configuration" for BIB, one that could, theoretically, be supplied separately from a container.

@cgwalters osbuild/bootc-image-builder#18 (comment)

Right now, the scope of what's in bootc as far as partitioning is tiny - just the root
filesystem type. And I think that's by far the most important thing.

So in the end if bootc will be able to produce a partition table for BIB

Let's not try to do that in any kind of near term; it sounds complex.

I guess in some order of priority I'm thinking:

• Root filesystem type
• Root filesystem max size (edit: although actually, because we don't ship a default growpart unit in the base image, anyone deriving from that will have extra space by default)
• tpm2-bound LUKS
• NBDE/tang

@cgwalters osbuild/bootc-image-builder#18 (comment)

... let's narrow in on supporting configuring the rootfs in the container image, because that was already extant before and is IMO by far the most obvious one to configure and most impactful to the user/admin experience.

@achilleas-k osbuild/bootc-image-builder#18 (comment)

we shouldn't oversimplify and think ahead a bit. At some point, our configuration structures will have to become stable. I don't think we're there yet, but we will be soon. So I'd like for us to have a good, high level view of where we're going. If we plan for the container partitioning metadata to grow, a simple flat json file wont cut it, so we might have to start thinking about the structure of that data now.

---

I tried to pull some of the most important parts of that discussion for some context. I hope I didn't misrepresent something by pulling it out of its original context.

What I would love to get out of this discussion is a plan for where we want to end up for the partitioning features and how we can get there from where we are now. Questions:

Overall, how do we imagine the disk creation flow starting from a base container, that may or may not get extended by a user into a derived container, and then through BIB become a bootable disk image? What part of the partition table is defined in every step and where do we define fallbacks/defaults?

• Will we have a default partition table defined as part of the base container metadata? Is it a full partition table, an enum of certain predefined configurations, or something simpler (like only root fs type)?
• Will users have the ability to define every part of the partition table in their derived containers? If so, what will that look like? BIB will have to consume this, so it's important to think about what it will look like.
• Will users be able to sideload partition table configs during disk image builds to override the container defaults?
• Will we support building disk images from containers that have no PT defined, falling back to defaults built into BIB?

[cgwalters]

One topic threading through this is whether we aim to support the the "dd raw disk image to metal" for bootc installs. I personally am trying hard to back away from that because of my experience from the CoreOS side is that while it covers extremely well the 85% cases, and causes deep problems in the harder ones like iSCSI, multipath, etc. Also the LUKS thing is just ugly where we need to have a "dummy" LUKS header today. And finally, another corner case is "advanced format" disk drives which are only going to increase in importance - and needing to juggle two raw disk images just plain stinks, even though we did some clever optimizations on the coreos side.

It is very much this experience that motivated the design of bootc install, where (like Anaconda) the partitioning and filesystems are computed on the target device.

And this again leads to an important point to me that it's not just "bib and bootc-container" - it's "bib and anaconda and bootc-container".

[achilleas-k]

I guess again I am just not understanding the scenario in which we need osbuild to setup LVM devices or do anything nontrivial - it seems like that's Anaconda's job (or in reality, we implicitly support the user doing whatever they want, and bootc install to-filesystem on it).

I don't know why someone would want it, and I have no idea how widespread it is, but people can (and afaict do) use LVM in the cloud (public or private). We can decide whether we want BIB to be able to do that or not, but we have the code for it.
I believe it was originally added in osbuild and osbuild-composer for edge bare metal builds and simple provisioning (no Anaconda), but now it's automatically used in cloud images too when users customise the default partitioning.

I guess...hmmm...if as an OS builder that wanted to deploy on metal, and default to LVM but still test in the cloud...

I can also imagine people just wanting uniformity across environments. Knowing they can maintain and admin all their instances, metal or cloud, in the same way.

probably we need a workflow where I can embed either a kickstart/blueprint like thing in the container (or pass to bib too if I prefer) and have that apply when making custom disk images that I upload to the cloud for a baseline for testing.

This is where I was going with the initial recommendations and some parts of this comment. We can design whatever format we want for bootc to communicate partitioning to BIB. My suggestion would be to build on the blueprint (add filesystem types, lvm, luks) and make a format that covers that, but that's mostly because I also want to have the same capabilities in osbuild-composer (and we will, eventually). I feel like the Kickstart command/procedural style doesn't fit well with a config of this type but that's a design decision and we just need to make a decision.

[cgwalters]

I don't know why someone would want it, and I have no idea how widespread it is, but people can (and afaict do) use LVM in the cloud (public or private).

It really comes up in "pet" or even semi-pet instances where reprovisioning may be expensive and one wants to expand things online. In a cloud it's super easy to just attach a new EBS-or-equivalent disk, but to add that to your root or /var/lib/postgres or whatever you really need LVM (or btrfs, or stratis).

[achilleas-k]

Yeah, so it's basically up to us: Do we want to support it in bib for cloud images or not? I think we should but I'm fine either way.

[cgwalters]

Yeah I think we should support both LVM and e.g. Stratis etc. at some point in bib.

However, both probably need to go below other things, I think LUKS on metal is a lot more important for example (which turns into the kickstart-configuration question).

[jlebon]

xref: coreos/fedora-coreos-tracker#1646 (comment)

[cgwalters]

One thing I could imagine doing here is that we support in bootc something like a single entrypoint /usr/lib/bootc/install/prepare-disk that can be injected into the container image (and the current Rust logic there could be split out as a separate example entrypoint), but we could even take the Go code that exists in osbuild and compile it into the container image (there's some mechanical ugliness with this I'm sure) - but it would at least get least get us closer to a place where (once anaconda runs bootc install) in theory we get closer to having the same partitioning across cloud images and metal installs.

In this we entirely avoid the need for some protocol between bib and bootc (except for the problem of osbuild wanting to have the manifest "on top"), and also it's not exposing to farther-downstream users any kind of fully general partitioning configuration mechanism (except of course that one already exists in kickstart).

[cgwalters]

Another open question: As I understand it, you want bootc to be doing the partitioning in the future (running fdisk, mkfs, etc) but you don't want every part of that partition table description to be contained in the container (correct me if any part of this sentence is wrong).

No, I don't think bootc always needs to be doing the partitioning. Supporting bootc install-to-filesystem is very much aimed at supporting use cases where external things do partitioning. The trickier detail in that though is just ensuring that the container preferences (unless overridden by disk image generation time preferences) are honored.

A good rule of thumb IMO for whether something should be in bootc (versus disk image generation or somewhere else is):

• It needs to be managed in a consistent way both at install time and for subsequent updates

We know we can't change partitioning by default across in-place updates, so it's not bootc's core concern.

That said at the same time the use case of an OS carrying its own baseline installer is important to me because it's just incredibly useful, and neither osbuild nor anaconda are today really suitable for that (anaconda is just quite large).

[achilleas-k]

One tangential thing related to this is it'd be nice to not completely hardcode the /boot/efi path because systemd has been pushing for just /efi and I think that makes some sense.

That's just a decision we (or, well, you) can make at any time and we can change it though. The fact that it's in code and not some json or yaml file that we would have to build into the bib container anyway isn't something I see as an issue.

Although I guess to align this with the existing osbuild [[customizations.filesystems]] we'd have to have a mapping between mountpoint and these system partition types.

It would be nice if they had the same structure but I understand if they need to deviate. I don't think it'd be much work for bib to map these. There is one decision we can make as to whether we want the user supplied customizations to look more like Blueprints or more like the bootc install print-configuration output. The latter makes sense for tighter integration, but the former gives us the ability to reuse Blueprints and all the other customizations, when we decide to expose them.

I would really say the default expectation is to use a local image from containers-storage - so we can easily just run the container, right there by default. All the gyrations around copying via skopeo to a dir, then copying again to containers-storage, making another copy of that filesystem tree via cp -a, are just unnecessary in this model.

Understood. Let's make this a requirement then. The local container store access is almost done so we can build on that to never have to copy a container and just use it from the store directly.

No, I don't think bootc always needs to be doing the partitioning. Supporting bootc install-to-filesystem is very much aimed at supporting use cases where external things do partitioning. The trickier detail in that though is just ensuring that the container preferences (unless overridden by disk image generation time preferences) are honored.

My thinking on this is that anyone (person or project) that uses bootc install-to-filesystem should know that a bootc container has a preferred partitioning scheme. Any project that consumes a bootc container to build a disk image should follow what's printed by bootc install print-configuration or deal with the side effects. In the case of bib, we can follow this rule as strictly as we know is needed so, for example, we can be strict about following the esp option, no exceptions. If anyone's building disk images by hand and using bootc install-to-filesystm, then it's on them to do what's needed.

Or to put it another way, if bootc install print-configuration defines a root fstype and size, an esp fstype and size, etc, we can let users change the root fstype and size (which shouldn't be a problem?) but not allow modifying the esp size (or only allow growing it) because that's tied to a requirement of the base image.
We can figure these rules out as we go, that was just a quick example, but I think it's fine, in principle, to be as strict as necessary with this ourselves (so we can guarantee sane disk image builds) and expect that anyone else doing things manually will have to know what's needed.

[achilleas-k]

That's just a decision we (or, well, you) can make at any time and we can change it though. The fact that it's in code and not some json or yaml file that we would have to build into the bib container anyway isn't something I see as an issue.

To clarify, I'm not saying it's not important. Just that we can work around this in our current model (of defaults being in code) with very little disruption or effort. If we want to externalise it somehow (along with other parts of the default/built-in partition table), we can talk about what that would look like in another thread.

[cgwalters]

My thinking on this is that anyone (person or project) that uses bootc install-to-filesystem should know that a bootc container has a preferred partitioning scheme.

I would say that in the general case there's actually multiple possible preferred, or variants thereof. The most obvious example is supporting LUKS versus not. For a given generic base image as we ship today, some users in public cloud won't want to pay that extra cost because they believe the IaaS handles it for them. On bare metal, it's usually a requirement.

(It's very tempting in some ways to try to default to tpm2-bound luks on by default actually but...)

There's huge amounts of possible complexity here, and I can't even imagine that we try to define some sort of generic format for possible preferred schemes.

I guess I'm just saying again I don't think it's we're trying to have the container image hardcode a single partitioning in general versus providing some defaults for particular portions.

[achilleas-k]

Right. But my main point here was that we can have bootc provide information, communicate the config that was built into it, and each consumer would need to decide if it follows it or not, but we on the bib side can be more strict about what we allow or not (like enforcing a minimum size for esp when the container defines one).

[bcl]

I seem to be missing something here. Why does the container (which may or may not have any expectation of how it will be run) have any filesystem or partitioning information in it? I thought the whole point of this process is to take a container you're currently running with podman and turn it into a disk image you can use on your hardware.

[cgwalters]

The person making the container is making an operating system for which they want to set defaults. For example in my case, I maintain an image for which I expect the default filesystem to be xfs, not ext4.

Sure, we should support dynamically overriding that choice when making a disk image, but the only really sustainable thing in any kind of general case really for the default to come from the container which defines the operating system.

The secondary thing is that in the bootc world we want the source of truth for operating system configuration in general to be the container image, not at disk image time. For example, one can also inject default install-time kernel arguments via bootc just by adding a file into the container build - not forcing everyone materializing a disk image to duplicate that logic.

[bcl]

I went back and looked at the bootc docs again and figured out why I was confused :) These are special bootc containers, not generic containers as I was thinking. So I now see why there may need to be multiple partitioning 'layers' involved.

I think it does need to be able to override previous settings at each layer. For example, the final output from bib may be specific to an image type (eg. an iso) that requires special treatment. The creator of the bootc container may not have any idea where that base will end up being used. But intermediate layers may require specific partitioning for their application (database storage?) that shouldn't be overridden.

Maybe we need to mark things as optional / required? And if a lower required setting is overridden the creation fails with an error.

[cgwalters]

The bootc containers can be run as a container, and must be runnable that way to be part of a container build pipeline. But indeed they are intended to be deployed on a virtual/physical host.

[cgwalters]

Let me move this back to the top...from my PoV, bootc does not need to (and really can't in general) define its own comprehensive partitioning schema; it's in the end operating system specific.

So one thing we could do is say that for Fedora derivatives using bootc via bootc-image-builder, we will support injecting /usr/lib/bootc/install/blueprints or something.

I guess hmm...what probably needs to also happen in that case is that bootc install should basically error out and say that an external installation tool is expected to be used. In that case then, bib could extract the blueprints from the image and use them directly.

---

(I know the content-in-image thing brings back up the "manifest over image" thing...and it makes sense to me to support also providing them externally. It seemed though like we were reaching some agreement that we'd always have the container image in containers-storage or mounted in some way, so we can easily execute/introspect it. I also still don't quite understand in what cases we really want to generate a whole temporary filesystem tree that we only then copy to a disk. That still feels to me like a "debug only" thing, and so if we switch to writing content to the disk directly, it's easy to have a flow like:

• mount/run container to extract /usr/lib/bootc/install/blueprints and other config
• create disk according to embedded schema
• Install container to disk

[cgwalters]

bootc does not need to (and really can't in general) define its own comprehensive partitioning schema

(It'd be really nice not to carry a fully hardcoded one upstream, which is a distinct but related thread)

[achilleas-k]

I know the content-in-image thing brings back up the "manifest over image" thing...and it makes sense to me to support also providing them externally. It seemed though like we were reaching some agreement that we'd always have the container image in containers-storage or mounted in some way, so we can easily execute/introspect it.

I think we all agree that we can work on getting this working; run the container to get the config and prepare the build. Right now, I don't see any reason to move away from the "manifest over image" situation, but we can find out quickly if we have to.

That still feels to me like a "debug only" thing

It's also a good "separation of concerns" situation for me. As in, it's a bit simpler to work on a tree that's just on the local filesystem, assemble everything needed, then just put it onto a disk. In the classic case, given how osbuild works (each stage is set up and torn down separately), mounting a disk to perform a single file write for a single stage then doing it again is inefficient. But that's a different story that doesn't apply here. It's like that here because we're carrying over the way we generally work because we have predefined pipelines that fit the requirement.

• mount/run container to extract /usr/lib/bootc/install/blueprints and other config
• create disk according to embedded schema
• Install container to disk

This should work. Especially when we start using bootc directly.

bootc does not need to (and really can't in general) define its own comprehensive partitioning schema

(It'd be really nice not to carry a fully hardcoded one upstream, which is a distinct but related thread)

I keep coming back to this but, there has to be a default or standard partition table somewhere for fallback. If we don't need one in bib in the end, if it can come from somewhere else, that's fine.

[jlebon]

Chiming in from the CoreOS side here.

Work is underway to switch over to osbuild soon in our pipelines for some of our streams. On the user side, as a first approach they'll be using osbuild directly + mpp files just like we do in the pipelines to generate their custom images. Longer term, obviously we'll have Composer/Image Builder integration but we'll also want something nicer on the CLI (that's more streamlined than Composer). It's still not clear yet whether bib is that something, or if we'll want something else. One of our goals I think should be to use the exact same tool in our pipelines that the user could use to generate disk images.

That said, at the end of the day all the goodies will be in osbuild/images anyway, and ISTM disk generation tools should just be semi-thin wrappers around that focused more on the UX than the contents of the resulting artifacts. Notably, semantics like what's discussed here (config files in the container controlling disk image generation) basically need to live somewhere lower in the stack because we would want it to work as part of the Image Builder path too, right?

Anyway, specifically on the subject of partition tables and rootfs type, some thoughts from my side:

• it makes a lot of sense to have users customize some partitioning and filesystem things via files in the container
• currently, the language of partitioning in CoreOS is Ignition, and I think a natural fit here would be for users to be able to bake Ignition config snippets in their container images (e.g. /usr/lib/ignition/...)
  • as a first pass, all we would need to do is put the configs in the right directory in /boot so that they get picked up and passed to Ignition on first boot (that mechanism already exists, but doesn't support multiple Ignition snippets yet)
  • down the line, we can look at converting the partitioning and filesystem sections to a disk.PartitionTable object that would then be merged with the base definition baked in osbuild/images; that would allow the customizations to be pre-applied (and in the case of root changes, avoid the cost of first boot reprovisioning)

Obviously we could support blueprints too as suggested here, but... IMO the story is more compelling if users can reuse their Ignition configs in this flow.

[cgwalters]

It's still not clear yet whether bib is that something,

This is getting a bit tangential but I suspect an ergonomic problem here is that having bib code as its own isolated container and not a reusable binary will break the current coreos pipeline model of having a single pod do a lot of things. Arguably though it would be better to schedule multiple pods that share a transient volume.

[achilleas-k]

Notably, semantics like what's discussed here (config files in the container controlling disk image generation) basically need to live somewhere lower in the stack because we would want it to work as part of the Image Builder path too, right?

Yes. If the partitioning handling becomes something we want to include in image builder, it will be moved to osbuild/images so that it can be reused in osbuild-composer (and by extension the image builder service).

[cgwalters]

Notably, semantics like what's discussed here (config files in the container controlling disk image generation) basically need to live somewhere lower in the stack because we would want it to work as part of the Image Builder path too, right?

Well that seems a bit tricky because IMO it opens up a bit of a can of worms around the general issue that today Image Builder (the existing composer/service/etc) tries to treat "package based" and "ostree/container based" as equal things, with effectively it on top, and these two are more "format" backends.

The bootc idea is much more that the container is on top - the driver, the north star, the source of truth, the technical heart. And that's why bootc defines a basic config file to for things like the root filesystem, because that is an important configuration knob and there is really no sane default one can apply without trying to introspect the OS version.

This really shows up in that the container image provided as input to the system (for bootc-image-builder, the new thing here), defined externally. And a related thing to that is that a container image is a fully formed artifact, unlike a pile of RPMs. In theory of course, absolutely nothing stops one from creating an RPM with partitioning information, but it'd be awkward and weird to do it.

My inclination here is not to try to maintain that "Image Builder parity" approach going forward because it will create more confusion and technical problems than it solves. Let's just keep the "build an AMI from some RPMs with blueprint tweaks" etc. working as is.

[jlebon]

To clarify, when I say "Composer/Image Builder integration", I'm still talking about the user passing in a container image ref and getting out disk images, not composing a system from scratch.

I think it'd be really surprising to have only some features work when using one flow but not another to accomplish the same thing. This stuff should really be defined in a shared space.

Yes, this in theory also allows having an RPM ship those same files, but that's not really what we expect users to do nor what we need to optimize for.

[cgwalters]

Filed containers/bootc#287 related to some realtime discussion today