The staging plugin is used to enable the new factory workflow to work with staging projects. This plugin implements some functionality that some day will be directly implemented in OBS itself. Meanwhile this plugin is used as a playground field to explore these new concepts and operations.
This plugin is used by the release manager to move requests into staging projects to evaluate the behavior and inter-compatibility of different changes working together.
An staging project is a mere project linked to a subset of Factory (MinimalX) , but with some packages that are different as a result of one or several requests. When the release manager assigns a request in a staging project, this request is review inside the project.
There is a precondition here, the user that has the role of staging manager needs to have permission in OBS, and they need to be:
-
Maintainer of openSUSE:Factory:Staging (needs current staging manager)
-
Group member of factory-staging (needs OBS admin)
Clone this repository:
git clone https://github.com/openSUSE/openSUSE-release-tools.git
Link osc-staging.py and the library either to ~/.osc-plugins or
/var/lib/osc-plugins
cd openSUSE-release-tools ln -sr ./osc-staging.py ./osclib ~/.osc-plugins
Optionally, one can create directories for each project of interest in order to
alleviate the need to use the --project option for each command. For example:
# Or use a directory already initialized by osc. mkdir -p "openSUSE:Leap:42.3/.osc" cd "openSUSE:Leap:42.3" echo openSUSE:Leap:42.3 > .osc/_project # runs against openSUSE:Leap:42.3 osc staging list # runs against openSUSE:Factory osc staging -p Factory list
The following snippet may be useful for addition to a bash PS1.
if [ -f .osc/_project ] ; then echo " <<$(head -n 1 .osc/_project)>> \$ "; fi
This provides a clear indication of the project against which the commands will run and can be merged with an existing complex PS1.
This plugin is expected to be used interactively. The basic commands are:
-
accept Accept all requests assigned to a staging project.
-
check Check the status of the different request inside the staging project, including the status in openQA.
-
cleanup_rings Check the status of the different rings in Factory.
-
rebase Freeze the packages that are not affected in a staging project. In the past, this command used to be called freeze, which is still a valid alias.
-
list Accept all the requests from packages that are not in any ring, detect request that supersede the current ones that are actually tracked in staging projects, and list the remaining requests.
-
select Move a request (or a list of requests) to a staging project, or move requests between staging projects.
-
unselect Remove a request from a staging project, and publish it again into the backlog.
This action will list all the requests that are candidates to be assigned to a different staging project. To decide this list, this command will perform two actions with side effects into the requests status.
The first action is to accept requests that do not affect packages that are not inside one of the rings that are defined in Factory. This acceptance will move the request to the next status of revision (maybe waiting to the manual review or the legal review).
The second action is to detect if a request supersedes a different one that is currently in a staging project. If this is the case, this command will replace the old request by the new one, declining the superseded one after this.
Because of the different side effects, this command needs to be executed with the proper permissions in OBS.
The release manager or the staging project manager can take a request
and assign it into a staging project using the select command.
Actually we have ten staging projects, that are managed according to some internal rules. Every time that the manager decides to move a request to one of those projects, a metadata attached to these projects is updated, and a new comment is generated.
The metadata associated to a staging project contains the list of requests that are assigned to the project, and is used by the plugin to track and manage the different requests. For example, the command:
osc staging select A 12345
will add the request #12345 into the staging project
openSUSE:Factory:Staging:A. This command will take care that this
request was not previously in the staging project A or in a different
one. If this is the case, it will show an error message and exit without
selecting the request. If we want to move a request from one staging
project to another one, we need to use the parameter --move:
osc staging select B --move 12345
This command will move #12345 from openSUSE:Factory:Staging:A to openSUSE:Factory:Staging:B
You can also merge staging projects by giving other staging projects as arguments:
osc staging select B --move A C
This moves all requests currently staged in A and C into B
The unselect command will untrack a request from a staging project,
returning it back to the original backlog (without approving or
declining the request) So for example, if #12345 is being tracked in
A, the command:
osc staging unselect 12345
will find the correct staging project and remove the request from it.
Before accepting the requests inside a staging project, the user can
check the state of those requests. The check command will check the
project status, taking care of superseded requests or already accepted
requests.
This command will also check the status in openQA of the project.
This command can be called without special permissions.
If the current status of the staging project is good, this command will change the review status of the different requests assigned to the project, accepting them.
Internally, the accept command contains a call to the check
command to make sure that the request can be accepted.
After this command, the staging project status will be disabled, to avoid the overload of OBS.
Rings are collections of packages that are deeply interconnected, and that are building basis for a different ring or for the rest of the distribution. A ring is a way to organize Factory into inter-dependent packages that are again used to build a different layer of Factory itself.
Actually we have identified three rings:
And you can find a better description in this very old presentation from a conference celebrated in 2014 in Dubrovnik, Croatia.
This command is used to check the current status of the rings and to find undesirable dependencies of the packages that conform the rings.
Factory (or the subset MinimalX) is always a moving target, even with the staging projects. If we want to check the status of the request assigned to a staging project, sometimes it is desirable to have a frozen status of the source packages that are part of Factory but not of the staging project.
This command is used to build frozenlink-kind-of links for a staging project.
The staging manager wants to see the status of Factory every morning.
osc staging check
With this status they will contact the author of the request, or will rebuild the packages if there is a suspect or a random fail.
After that, the staging manager can check the important packages that are pending in the queue.
osc staging list
This command can update the request attached to a staging project, replacing the superseded one. The list shows the name of the ring where this package is found.
The list command shows the name of the ring where we can find the
package involved in the request. This information is important,
because actually only the staging project A supports these kinds of
packages.
osc staging select A 12345
This command will put the request #12345 into the staging project A. If A is full, the user can wait until A is empty again before putting new packages from Ring:0.
The staging manager wants to move some packages into different staging
projects. The complex part is to decide how to distribute the
packages here. The staging manager needs to make sure that packages
that have related changes (e.g. new rpmlint check and the packages
having fixes for it) are tested in one letter.
osc staging select B 22221 22222 osc staging select C 22223 osc staging select B 22224
The select also has a --move to correct mistakes done on first run.
osc staging select --move C 22224
Also the staging manager can return some request into the original queue.
osc staging unselect 22224
Staging projects should not be too small, but not too big either - and staging projects that are almost done testing shouldn’t get a re-triggered build. So in practice adding them in large batches has proven useful, i.e. adding to B for half a day and then open up C and add to it from then on and only look back at B if there is a problem.
From time to time the staging projects need a rebase to make sure that
they are still working with the current status of Factory. For this
the staging manager can use the freeze command to update the links
of the packages.
osc staging freeze
Of course, this will be done only when the project is in green status and Factory, the base, is also green in openQA. In other case we can see errors in the staging project that comes from Factory.
A pro-tip: the ring projects should be basically built and tested - quite challenging to find the right moment.
Checking the current status
osc staging check
In the list we found a request that is independent, we moved it to a isolated staging project.
osc staging select B 12345
There is also a Ring:0 package, that needs to be in A
osc staging select A 12300
Also we found three YaST packages that are related.
osc staging select C 22201 22202 22203
We wait a bit and we check the result in openQA. We see that the packages work properly in OBS (compile correctly), but there is something wrong in openQA: some of the tests are failing.
In this situation we can:
-
Rebuild the image in openQA to see if this is a random problem.
-
If openQA is red again, check the packages that can be problematic in the staging project, maybe reading the changelog.
-
With this information, remove one of the requests from the staging project, putting it back to the queue.
osc staging unselect C 22202
After a while we see some packages that are failing in OBS, we need to discard that is a random fail, we re-trigger the build:
osc rebuildpac $PROJ $PKG $REPO $ARCH
From time to time, we see that there is a missing dependency for one
of the packages that is in the staging project, and this dependency is
not in the subset of Factory (MinimalX) that is linked in the staging
project (for example: ImageMagick needs libqr to build properly). In
this case we need to linpack this package into the staging project:
osc linkpac openSUSE:Factory liblqr openSUSE:Factory:Staging:F