Standard tool/workflow/pipeline for auto-landing Fedora dist-git PRs

[martinpitt]

Thanks to the current packit tooling and service, downstream release automation has come a long way. There's just one remaining item: Fedora dist-git PRs for new releases have to be watched and landed manually. This is still a regression from what we used to have in Cockpit (in exchange, we get gating tests before landing, which is good!).

To be clear: I do want to keep having Fedora PRs instead of direct dist-git pushes. They are a great place to document the update, collect/discuss gating results, etc, and packit service should better not have an almighty Fedora token that is able to push to all dist-gits. What I'm looking for is some standardized tool/workflow/pipeline that project authors can use to automate this step without too much fuss.

The osbuild team has come up with https://github.com/osbuild/fedora-bot , which provides the necessary glue for checking for open PRs, waiting for the tests to succeed, and landing them. It needs a token from the developer which is privileged to land that package (and we should keep that trait). However, it is hilariously expensive. It should probably be changed to start a polling loop when a tag is pushed, wait for the PR to succeed, and land it. That may run for an hour or two, but at least it doesn't start hundreds of VMs in the two weeks when no release happens. 😁

Perhaps there is even some way with doing that in an event-based way? UMB or so? I figure pagure doesn't make this easy, and we may need to wait for migration to GitLab to happen.

In Cockpit we also use fedora-bot, but not through a workflow -- we just call it manually. This is also not optimal, as there are only two people in the team who are Fedora packagers, and one of them (me) is not even formally in the team right now 😁

Has this been discussed anywhere already? Can we provide some tool, standard github action, or at least some documentation in packit how to achieve that in a sane way?

CC @ochosi

[martinpitt]

NB: This does not necessarily have to live under the packit project -- now that I think about it', it's much more of a generic Fedora project than packit specifically. But since you are so keen on automating the Fedora releases, you may at least have some good input on this. 😉

[TomasTomecek]

Martin, thank you for opening this! We have discussed it within our team, I think that Franta will provide a more detailed update :) if I recall correctly 😁

I have also submitted (not just) this for the next OSCI focus group: I'd love to discuss our strategy around automation so we all can coordinate.

[lachmanfrantisek]

Here are the promised details as a result of the team discussion:

tldr: We don't want to do this as Packit but we can help.

• This is not a technical issue, but more an agreement/legal one. We, as Packit, can do that but don't want to be the gatekeeper and have all the responsibility for the content coming to Fedora.

  • One reason is to avoid having too much permissions (e.g. we want to get rid of dist-git commit rights completely in favour of a global Bodhi rights.)
  • If we can avoid being the gatekeeper, we can be safe that we don't do anything harmful (we can't break anything if push anything bad and on the other side, we can be safe when building or creating updates for anything that landed in dist-git.
  • Ideally, there should be at least one step done by a real user.

• But, I see that there are use cases that one might want to merge PRs automatically:

  • E.g. There is a set of PRs for all the Fedora versions with the same diff and approving just one might be enough.
  • There is a clearly defined set of conditions needed to be fulfilled before the merge is performed. (E.g. all the checks passed and the author is somehow approved to do this.)

• So, we can at least help and/or make that step smoother:

  • We can inform people about the dist-git PR or do anything else around that PR to save some time/clicks.
  • We can link/suggest another tool for this step. (Do you want to market fedora-bot?)
  • We can help with the other tool (i.e. fedora-bot) to do this.
    • Regarding the implementation, we already have a library for working with Pagure and already listen on fedora-messaging to be notified about dist-git events (what can replace the polling). So we can steal parts of the code.
    • Also, we need to discuss/plan how to support new projects (every project can set up its own instance or we can have a global service for all the dist-git projects,...)
  • Alternatively, can't we make Fedora zuul do this?

Tomáš is right, it would be nice to meet and discuss this.

[martinpitt]

We, as Packit, can do that but don't want to be the gatekeeper and have all the responsibility for the content coming to Fedora.

Fully agreed. That's exactly what I spelled out in the description -- this is about coming up with an agreement and tools to do this, but the responsibility should stay with the development teams -- i.e. they would each create their own Fedora pagure bot token to "press the button".

E.g. There is a set of PRs for all the Fedora versions with the same diff and approving just one might be enough.

TBH, this feels too special-case-y and complicated to me. The set of conditions for auto-merging should be applied to each PR individually.

We can inform people about the dist-git PR or do anything else around that PR to save some time/clicks.

That already happens, dist-git package maintainers get an email about opened/closed PRs.

we already listen on fedora-messaging to be notified about dist-git events

That is indeed the sole blocker here. fedora-bot exists, it's easy enough to set up and use, but right now there is no suitable event provider/trigger to run it, other than dumb polling. I.e. if there is any outcome/agreement/solution about this problem, then please let it be this. (I don't have anything in mind, I don't know how pagure can be listened to/automated). Hence my gut feeling that this might have to wait for Fedora moving to Gitlab -- with pipelines this would be totally straightforward.

[lbarcziova]

Output from Packit 2023/Q1 planning meeting:

• What's the benefit?
  Packit would offer complete automation of the Fedora downstream release pipeline

• Who will benefit?
  cockpit, potentially other projects using Packit in downstream?

[TomasTomecek]

Sooner or later, we should start a discussion in Fedora community about this.

[lsm5]

got sent here by @nforro . I / container-tools team would love to have this feature as well. Haven't yet played with fedora-bot but I can give it a try. But would be really cool to have it done in packit itself.

RE: responsibility, if this can be done via the package admin explicitly adding fas:packit to the package as a committer, so it's finally on the package admin, I'm totally fine with that.

RE: having a human do at least 1 step, how about the bodhi creation having autopush to stable disabled on both karma and number of days? A human has to monitor bodhi gating tests anyway, they can also monitor karma and stable push.

[lachmanfrantisek]

After a team discussion, we didn't pick this as a top Packit team priority for the next quarter and preferred epics with bigger user benefits. (Sadly, we have limited resources...) If anyone wants to help us with this (e.g. start the discussion in the Fedora community), we will be really glad. We are open to any collaboration and have already successfully implemented/started multiple affords thanks to people outside of the Packit team.

[martinpitt]

With Fedora moving to Forgejo we finally have a much better perspective: We could add an action to dist-git which auto-lands (or at least auto-approves) a PR once the tests are green.