mixer CLI for setting sources without downstream/online/remote orientations

[gtkramer]

Using the mixer CLI as an upstream is somewhat difficult due to the assumption built into mixer that the user is a downstream. This difficulty is mostly encountered when trying to define the sources (bundles and repos) for mixer. The commands that are used as an upstream work around the downstream assumptions and make the code that DevOps writes more difficult to understand because it works around unexpected behavior. What's outlined below is just a brainstorm of what might help to solve this problem.

mixer init

Sets up a workspace by creating a directory skeleton, creating mixer.state with a default format, version, and last version, and creating a default builder.conf. Only does the work that is missing, and the values of existing files are not modified. If all of this already exists, then this command is a noop. This command may, however, perform a conversion of mixer.state and builder.conf to a different format if necessary. If other changes in the workspace structure are required to be converted, this would be when those changes are made. This command would always be run and would not take any arguments. No other work is done with this command.

mixer config set Swupd.CONTENTURL "${DISTRO_URL}/update"
mixer config set Swupd.VERSIONURL "${DISTRO_URL}/update"

Next logical step in the process. Same as what we have today.

mixer state set Mix.FORMAT 1
mixer state set Mix.VERSION 31290
mixer state set Mix.PREVIOUS_MIX_VERSION 31280

Uses any of the values supplied from the command line to modify mixer.state. These commands would be run as required by automation. No other work is done with this command. Mimics what is done with the mixer config set subcommand. Format would be automatically modified during automatic format bumps. Last version would be automatically modified on successful creation of an update. Version is not automatically modified except by the user at the mixer CLI with this command. Any time mixer.state receives an additional key-value pair, it needs to be exposed for modification through the CLI to provide an interface for automation.

mixer repo sources 31290 file:///home/releases/www/31290/clear/x86_64/os
mixer repo sources 31290,https://clearnas.ostc.intel.com file:///home/releases/www/31290/clear/x86_64/os
mixer repo sources file:///home/releases/www/31290/clear/x86_64/os
mixer repo sources /home/workspaces/clear/packages/koji/results
mixer repo sources /home/workspaces/clear/repo
mixer repo sources <location>...
<location> = build number,mirror host
<location> = curlable path
<location> = local file path to rpms outside mixer workspace (with or without repodata)

Creates any number of repo sources for mixer. Sources are layered from left to right, with sources having highest priority on the left and the lowest priority on the right (priority in the dnf sense). If a path is given that is not a valid repo, mixer copies the rpms into it's workspace and creates a repo automatically. Here, users would not know about local-rpm or local-yum folder. Using this command again wipes out any existing repo configurations in mixer's workspace. When it is run, mixer recreates the .yum-mix.conf again and its normalized repo workspace again. Required to be run every time prior to starting a mix. All sources must always be defined.

mixer bundle sources 31290 /home/clrbuilder/projects/clr-bundles/bundles
mixer bundle sources 31290,https://github.com/clearlinux/clr-bundles /home/clrbuilder/projects/clr-bundles/bundles
mixer bundle sources /home/clrbuilder/projects/clr-bundles/bundles
mixer bundle sources <location>...
<location> = git ref,git host
<location> = local filesystem outside mixer workspace

Creates any number of bundle sources for mixer. Sources are layered from left to right, where subsequent sources are overlayed on top of the previous ones. A downstream would do mixer bundle sources 31290 /home/clrbuilder/projects/clr-bundles/bundles whereas an upstream would do mixer bundle sources /home/clrbuilder/projects/clr-bundles/bundles. Here, users would not know about local-bundles folder or the mixbundles file. Using this command wipes out mixbundles file and any internal tracking of the existing sources. When it is run, mixer builds up the bundle sources again. Required to be run every time prior to starting a mix. All sources must always be defined. When running, if mixer sees that existing sources have already been cloned, it has the option of keeping them in tact. If users want to modify bundles, it needs done outside the mixer workspace and then they are brought in with this command.

mixer bundle add --all | <list of bundles> | --minimal

Pretty much the same as what we have today, except adding bundles no longer has the downstream vs upstream orientation and just asks the user, "What bundles do you want for this update now that you have defined your current sources?"

Something like this above would be all an OSV would need to do in order to get going with mixer, a possible BKM of sorts. There are a few goals that would be nice to achieve with a perhaps similar mixer CLI:

• Users do not need to be aware of the implementation of the mixer workspace and would treat it as more of a blackbox. Namely, users would not need to know about the local-rpms, local-yum, and local-bundles folders in addition to the mixbundles, mixer.state, and builder.conf files. Code that DevOps needs to write to use mixer would no longer manage these, or at least manage it less. Users come with their sources, and mixer consumes them. However, this would require users to be minimally aware of their sources, which isn't necessarily a bad thing, and isn't much different from how it is today.
• Allows mixer to be used for any number of downstreams, fully customizable for wherever the sources are. Can use default sources, or let users supply their own defaults when locations need to differ.
• The offline flag is no longer needed because it is up to the user to determine what is online or offline based on the sources they define. The same goes for local vs remote configuration because this is more of a property of the sources the user comes with and less about the configuration of mixer itself. Conditional behavior that happens during any mixer subcommand that keys off this offline flag is also no longer needed.
• Commands are not overloaded and management of the workspace is delegated to the respective commands.
• The distinction between upstream and downstream goes away because it is only relevant to how sources for mixer are defined, but that is now the responsibility of the user. Documentation would probably need to keep this distinction to help explain the concept to users. While it is an add for the user, it's not hard for the user to say, "I want bundles from 31290 first and then my bundles," since this information already needs to be supplied today and is what users typically understand.
• Mixer makes less assumptions for the user and puts the user in control of how they want to set up their sources.
• Reduce the number of edge cases in the CLI today where state is not being well-managed given all of the configuration options that are currently exposed.

I'm sure there are some more fine-grain details that I've overlooked given how the stateful nature of mixer makes this challenging, but I wanted to put the idea forward to see what other folks thought, and see if it might lead to a better mixer CLI experience for upstreams.

[gtkramer]

Similarly, I think it would negate the need for a separate tool for upstream vs downstream mixing since the differences are in how sources are defined. There is a good deal of functionality built into mixer for a use case that lets users start editing bundles with mixer, but I think this particular use case may be good for a separate tool to align better to the unix philosophy and simplify mixer's main code base a bit.

[gtkramer]

While the use cases where we'd actually need more than two sources for bundles and repos are probably limited, I think what we gain here is a more flexible interface for defining sources that is more straight-forward because it doesn't care about local vs remote resources, online vs offline resources, or upstream vs downstream users. There would just be logic to fetch the resources from wherever they're at and homogenize them into something that mixer can consume.

These distinctions add additional complexity to the mixer CLI and the underlying code. Given there is still some complexity in homogenizing sources, this may be something we want to require from the user's end so this complexity could be removed entirely. clr-distro-factory already does this with content.sh, which provides an interface to supplying input to mixer.

I guess the open question is, "Who is responsible for homogenizing inputs?" Mixer or DevOps? Mixer will know the format required for homogenization, and ideally this would be something that DevOps does not need to track as the format changes.

[gtkramer]

There is quite a bit of behavior built into mixer that checks whether it's being used correctly and sensibly. There's different checks for each of its commands. This is useful for users who are less knowledgeable about how mixer needs to work. However, I would expect this need to decrease with a CLI such as the proposed one above since users would go through less steps with fewer commands to set up mixer to build the next iteration of their mix.

For what makes sense to keep around, it may be useful to consolidate this behavior into a command that is optional for the user to run, something like mixer validate. This would help to narrow the focus other commands and keep their functions consistent with what the user expects without any additional behavior.

[gtkramer]

A mixer validate command may be confusing with mixer build validate. It might make sense to have mixer validate workspace and mixer validate update commands so validation operations are tied under a common subcommand of validate. This would also make sense for MCA because mixer build validate does not actually build content for consumption by the client.

[gtkramer]

Ideally, the mixer validate commands are optional since they are not a required part of the process. They would only give assurance to users that things are looking sensible for starting to mix and then after the mix finishes. If a user knows what they are doing, they should be able to confidently bypass these to get to producing and consuming updates faster. However, these are commands that we would want to run in any production environment.

[gtkramer]

The mixer validate commands would be run either just before or after the mixer build commands. Then, mixer does not attempt to perform validation as users are setting up their workspace.

[jwakre]

mixer repo sources 31290 file:///home/releases/www/31290/clear/x86_64/os
mixer repo sources 31290,https://clearnas.ostc.intel.com file:///home/releases/www/31290/clear/x86_64/os
mixer repo sources file:///home/releases/www/31290/clear/x86_64/os
mixer repo sources /home/workspaces/clear/packages/koji/results
mixer repo sources /home/workspaces/clear/repo
mixer repo sources <location>...
<location> = build number,mirror host
<location> = curlable path
<location> = local file path to rpms outside mixer workspace (with or without repodata)

Creates any number of repo sources for mixer. Sources are layered from left to right, with sources having highest priority on the left and the lowest priority on the right (priority in the dnf sense). If a path is given that is not a valid repo, mixer copies the rpms into it's workspace and creates a repo automatically. Here, users would not know about local-rpm or local-yum folder. Using this command again wipes out any existing repo configurations in mixer's workspace. When it is run, mixer recreates the .yum-mix.conf again and its normalized repo workspace again. Required to be run every time prior to starting a mix. All sources must always be defined.

I agree that the CLI should be improved to make it easier to manage an upstream mix and that all repositories should be treated equally (no special treatment for upstream/local). Although, I disagree with repeatedly requiring all repositories to be set on the command line because it adds complexity that most downstream users would find challenging. Alternatively, I think it would be best to initialize the mix with separate default configurations for upstream and downstream mixes. For instance, when initializing an upstream mix, upstream related settings would not be configured and when initializing a downstream mix, it would be configured to work with an upstream. Mixer should allow users that desire more advanced configurations to override these settings, but basic use cases should be as simple as possible.

I agree that it is a good idea to use a more generic approach (no special treatment for upstream/local) for bundle definition files.

Since most downstream users will want to use the upstream version to synchronize the repo/bundle, I think there is value to support variables. This will prevent users from re-configuring their sources.

Additionally, with these types of changes, we need to be careful that upstream format bumps will be crossed correctly by downstream mixes.