WIP: Repackage .deb builds as mozillavpn-beta #9820

oskirby · 2024-08-28T23:27:22Z

Description

We have had much recurring pain trying to upload release candidates to the staging APT repository, and it would be nice if we could find a better way to package and upload release candidates. This PR attempts to accomplish just that. We attempt this by doing the following:

We add repackage jobs to unpack a .deb file, edit its control file and repack it. There are two such jobs:
- repackage-deb-beta: Sets Replaces and Provides to mozillavpn and renames the package to mozillvpn-beta. This ensures that it will be considered an alternate version of mozillavpn as far as the Debian packaging world is concerned.
- repackage-deb-release: Changes the package version to the contents of version.txt, thereby stripping out the build suffix (eg: 2.26.0~build20241129 becomes 2.26.0) and we also add the version to the filename.
We add a transform that renames a taskcluster release-artifact to append a version number. This should allow the release jobs to pick up the repackaged .deb files rather than the one output by the build tasks.
We add a transform to the beetmover tasks so that promotions depend on repackage-deb-beta and shipping depends on repackage-deb-release.
We modify the beetmover-apt task so that it pushes to the release bucket for both promote and ship tasks. This was done by copying the in_production check from beetmover-promote and this is where I get a bit fuzzy on what the correct thing to do is.

Thus we wind up with a chain of tasks that should go something like this:

build-linux64/release-deb --+--> repackage-deb-beta -> beetmover-promote -> beetmover-apt
                            |
                            +--> repackage-deb-release -> beetmover-ship -> beetmover-apt

Reference

JIRA Issue: RELENG-1062

Checklist

My code follows the style guidelines for this project
I have not added any packages that contain high risk or unknown licenses (GPL, LGPL, MPL, etc. consult with DevOps if in question)
I have performed a self review of my own code
I have commented my code PARTICULARLY in hard to understand areas
I have added thorough tests where needed

After having a read through of the official Debian package format (see man deb(5)) it seems there are a couple of nuances to how the archive is created. We should probably be using the official tool, dpkg-deb, to manipulate packages to ensure that we create it correctly.

bhearsum

@jcristau should probably have a look at this at some point, as he's our resident debian expert.

I'm going to help get you set up with the staging repo to test this in any case.

bhearsum · 2024-12-02T15:18:24Z

taskcluster/mozillavpn_taskgraph/transforms/add_version.py

+
+    # If this is a beetmover promotion job, change the version to "beta" instead
+    if config.kind == "beetmover-promote":
+        version = "beta"


Does apt upgrade still work when the version doesn't change? (Do we care if it doesn't?)

This transform here is actually just changing the filename of the artifacts from mozillavpn.deb to either mozillavpn-beta.deb for promotion or mozillavpn-<version number>.deb for shipping. The version that apt sees is a value embedded in the package control file and isn't being changed by this transform. This comment is really referring to the string that we're going to tack onto the filename.

This may not have been 100% necessary to add, but I wanted there to be at least some ways to distinguish between the .deb files that came out of the build tasks and those that had been touched by repackaging, so I opted to go with a filename suffix like this.

ah, right, sure, that's fine

bhearsum · 2024-12-02T15:23:47Z

taskcluster/mozillavpn_taskgraph/transforms/beetmover_apt.py

+    is_production = (
+        config.params["level"] == "3" and config.params["tasks_for"] == "action"
+    )
+    bucket = "release" if is_production else "dep"


Just to be 100% clear: the idea is that both the promote and apt tasks will push to the exact same debian repo (but with different package names)? If so, this block looks like it ought to be fine. (We'll be able to verify the dep path in the staging repo, but the release one obviously can only be fully tested in a real release.)

So, my understanding is that beetmover-promote and beetmover-ship send packages to the production archive.mozilla.org site whenever they are triggered by shipit and we want to follow the same logic for the beetmover-apt task.

The promotion and shipping phases on archive.mozilla.org are distinguished by different directory paths (eg: promotion puts artifacts into the candidates directory). The objective of this PR is to distinguish promotion and shipping in the APT repository by changing the package name.

Are we talking about uploading to archive.m.o, the debian repository, or both? (Both are handled by beetmover, but they are distinct things can be done or not done independently AFAIK.)

jcristau · 2024-12-04T12:50:33Z

taskcluster/mozillavpn_taskgraph/transforms/add_version.py

+    cwd = os.path.dirname(os.path.realpath(__file__))
+    with open(os.path.join(cwd, '..', '..', '..', 'version.txt'), 'r') as fp:
+        version = fp.readline().strip()


The usual pattern is to use config.params["version"] instead of reading version.txt from transforms.

jcristau · 2024-12-04T14:05:01Z

taskcluster/kinds/repackage/kind.yml

+            command: >-
+                ./taskcluster/scripts/build/dpkg-repack.py ${MOZ_FETCHES_DIR}/mozillavpn.deb \
+                    --set provides=mozillavpn --set version=$(cat version.txt) \
+                    -o /builds/worker/artifacts/mozillavpn-$(cat version.txt).deb


I have some concerns about modifying the artifact in between promote and ship. That means if the repack script fails or otherwise introduces a bug somehow, you'll only catch it after shipping.
So my preference/advice would be for us to have 2 apt repos, one for release candidates and one for releases, and publish the exact same artifact to both (similar to how we have candidates and releases dirs on archive.m.o); that said, I don't actually want to block your work if this is the way y'all want to go.

jcristau · 2024-12-04T14:05:59Z

taskcluster/kinds/repackage/kind.yml

+            cwd: '{checkout}'
+            command: >-
+                ./taskcluster/scripts/build/dpkg-repack.py ${MOZ_FETCHES_DIR}/mozillavpn.deb \
+                    --set provides=mozillavpn --set replaces=mozillavpn --set package=mozillavpn-beta \


You'll want to set Conflicts in addition to Provides and Replaces.

oskirby force-pushed the deb-repackage-beta branch from 25b6c99 to 7424bb3 Compare August 28, 2024 23:56

oskirby added 4 commits November 26, 2024 12:13

Write script to repack and modify a Debian binary package

2449644

Add repackage job to rename packages to mozillavpn-beta

30ed936

Fix repackage dependencies

57b476f

oskirby force-pushed the deb-repackage-beta branch from 27ae26d to 33f3b18 Compare November 26, 2024 20:16

oskirby added 17 commits November 26, 2024 12:23

Try to test it?

9fa6569

Try to fetch .deb artifact

28c12b5

Add deb-release repack to strip RC version suffix

aaabc16

Oops, forgot to fetch

8d8972f

Append VPN client version to deb release artifact name

2efe7b8

I guess we have to rename in the beetmover tasks too

d4975f0

Once more with feeling

e660207

Try once more to fix the schema error

1ee256c

Lets try a little debug

7f5f16e

I think we have to explicitly mention the transforms

6c0ac99

Fix a path error

5e9ba97

Fix the path again

5e03e68

A bit more tweaking

881ce84

Simplify beetmover transforms

126b722

Tidy up some debug

4c61541

beetmover-apt should promote to release as well

ed92d3d

I guess we don't need the expicit dependency in beetmover-apt either

5b217c6

oskirby changed the title ~~Experiment: Repackage .deb builds as mozillavpn-beta~~ WIP: Repackage .deb builds as mozillavpn-beta Nov 29, 2024

oskirby requested a review from bhearsum November 29, 2024 23:13

bhearsum reviewed Dec 2, 2024

View reviewed changes

oskirby added a commit to mozilla-releng/staging-mozilla-vpn-client that referenced this pull request Dec 3, 2024

Squash upstream PR mozilla-mobile/mozilla-vpn-client#9820

bc6b1a8

jcristau reviewed Dec 4, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Repackage .deb builds as mozillavpn-beta #9820

WIP: Repackage .deb builds as mozillavpn-beta #9820

oskirby commented Aug 28, 2024 •

edited by jira bot

Loading

bhearsum left a comment

bhearsum Dec 2, 2024

oskirby Dec 2, 2024

oskirby Dec 2, 2024

bhearsum Dec 2, 2024

bhearsum Dec 2, 2024

oskirby Dec 2, 2024

bhearsum Dec 2, 2024

jcristau Dec 4, 2024

jcristau Dec 4, 2024

jcristau Dec 4, 2024

WIP: Repackage .deb builds as mozillavpn-beta #9820

Are you sure you want to change the base?

WIP: Repackage .deb builds as mozillavpn-beta #9820

Conversation

oskirby commented Aug 28, 2024 • edited by jira bot Loading

Description

Reference

Checklist

bhearsum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oskirby commented Aug 28, 2024 •

edited by jira bot

Loading