git-repo: Figure out testing for this monster

[jku]

Hardware keys, KMS keys, github actions... this seems like a nightmare to test but have to figure out something

[jku]

Some ideas from chatting with @asraa:

1. should aim to move things from the "GH actions layer" to "python command line tools" layer as much as possible.
   • this isn't something I was planning for repo tools especially but may be worth it for testing
   • Currently e.g. the git integration is not included in signer or repo tools. It would be useful for testing if it was as then you could run a semi-complete e2e tests within a (local) git repository
   • the signer would run in an environment with softhsm keys and the repo tools with maybe mocked online keys
2. the complete integration testing still sounds daunting but maybe repository_dispatch would allow setting up a separate test repository so that test workflows could be dispatched from repository-playground (or just on cron) the test workflows would simulate a user e.g. initializing the repo and would confirm that correct things hapen in git, issues and PRS? This might just be too complex...

[jku]

Looking at item 1 (moving more things into a python layer) more in detail, there are currently these "layers" in the whole system:

• workflows: these are part of user repository, come from forking playground-template
  1. signing-event
  2. snapshot
  3. version-bumps
• actions (signing-event, snapshot. offline-version-bump, online-version-bump) that contain
  1. typical sub action calls like checkout, setup-python, upload-pages-artifact
  2. potentially sub actions that require input like googleauth
  3. bash dependency install steps
  4. bash steps that run python tools and git
  5. javascript steps that interact with github api
• python tools
  1. playground-status
  2. playground-publish
  3. playground-snapshot
  4. playground-timestamp
  5. playground-bump-expiring

To make testing feasible outside of GitHub Actions I think there are a few possible improvements:

• the bash steps need to become executable outside the actions environment: either shell scripts or integrated into the python tools
• the signer tool needs to do a similar thing: git integration needs to happen either as external shell scripts or embedded into python. The goal is that playground sign <eventname> would pull the branch, present the changes, ask for signature, create commit, push and offer link to create a PR
• then the testing system could run repository and signer tools, simulating the GitHub workflow triggers and the javascript steps

One of my worries was that the "composite actions" are a design dead end and that I'd have to switch to docker actions but I don't think this is true: after the above changes I don't think docker action would be meaningfully easier to test: the limiting factor will be ability to actually modify the GitHub project, this will be equally difficult to test for all types of actions

[jku]

Documenting the current steps:

snapshot action

Following actions taken in multiple snapshot steps could be integrated into a single playground-snapshot command:

• run playground-snapshot
• if snapshot was created
  • git add
  • git commit
  • git push main
  • run playground-publish

online-version-bump action

This could be a single command "playground-bump-online" which does the same thing as current steps:

• run playground-bump-expiring snapshot
• if snapshot was not created
  • run playground-bump-expiring timestamp
• else
  • run playground-timestamp
• if snapshot or timestamp was created
  • git add
  • git commit
  • git push main
  • TODO: this should trigger publish just like snapshot does

offline-version-bump action

This could be handled by a command playground-bump-offline:

• for each offline role
  • run playground-bump-expiring ROLE
  • if new version was created
    • TODO git add
    • git commit
    • git push HEAD:$EVENT

signing-event

There are two separate bash steps:

• find fork point with git merge-base origin/main $GITHUB_SHA, store it for a javascript step and a checkout step -- I wonder if this and the checkout of the merge-base commit should be done in python so that this could be combined with next step?
• run playground-status, store the status for next javascript step

[jku]

Current status:

• repository tools (snapshot, bump-expiring)
  • expect current commit to be repository main HEAD
  • create commits
  • optionally push those commits to remote branch (main or signing event branch)
• signer tools (sign, delegate)
  • fetch event branch HEAD from remote
  • create commits
  • optionally push those commits to remote signing event branch

This should allow us to setup testing:

• create a git repository (by forking template)
• repository tools can run in this repository (without pushing to a remote)
• signer tools run in a cloned repository: remote is set to the file url of the real repository
• now it should be possible to alternate between repo and signer tools to simulate real workflows without doing actual network requests

[jku]

I'll try to come up with something this week