Add model signing unit test GitHub Actions

[mihaimaruseac]

No description provided.

[mihaimaruseac]

The token doesn't seem to be available when creating the PR to add the workflow, so that's why the CI fails.

https://github.com/mihaimaruseac/model-transparency/actions/runs/6628201742/job/18004764703 is the equivalent of this on my own fork (mihaimaruseac#2)

[laurentsimon]

Correct. We need to check how sigstore-python did it. iirc, they pre-generate some tokens every 15mn or so and put them in a bucket; then use these for the pre-submits. There's a lot of pre-submits we can add without signing zo, like unit tests for hashing, etc. Also, we will eventually need regression tests that ensure that previously-generated signatures can be verified by new versions of the verifier

[mihaimaruseac]

Yeah, I tried this as a quick way to check the dependabot PRs without needing to run manually or write longer unit tests. But probably it's better to revert to writing unit tests and running those instead.

[haydentherapper]

If we want to get something out, what about just having the tests run on merging to main and on demand so we can at least detect regressions?

[laurentsimon]

Can probably store a few signed models under a testdata folder and verify them on new PRs. At east that's test (1) the build / installation and (2) the verification.
Doing things on push works, but we need to handle automatic creation of issues when this fails, otherwise we won't know about the failures. It's do-able using something like https://github.com/slsa-framework/slsa-github-generator/blob/16fa37fa32d28bccf74a47327bcdfc4c47c47f5d/.github/workflows/e2e.upload-folder.schedule.yml#L174-L201 (need to create tags for labeling issues). I'm slightly worried about technical debt here. We're investing into something we're going to rip apart soon-ish. We're not changing the code atm, and the repo is clearly marked as experimental.
How about updating the pip dependencies for now and work on something similar to sigstore-python towards a proper v1 release (+ unit tests).

[mihaimaruseac]

That was my reasoning for sending the PR. We can also run locally (test model signing locally, run SLSA on own fork), but we'd probably still need a CI for dependency resolution to prevent #43

[laurentsimon]

So let's just test the installation then? Can be a pre-submit and is not throwaway work?

[mihaimaruseac]

How about updating the pip dependencies for now and work on something similar to sigstore-python towards a proper v1 release (+ unit tests).

Yeah, that's my impression after #43, we can just make a CI that tries to install the deps and if we think something might break further try to debug locally (like I just did for #37 and #32)

[mihaimaruseac]

So let's just test the installation then? Can be a pre-submit and is not throwaway work?

Yup, working on a PR for that