Add in-toto format with hashes of shards as subjects

[mihaimaruseac]

Summary

Note: This is an experiment serialization, one of the 4 in a series of PRs (#264, #265, #266, #267). Before a stable release of the library, we would standardize on an ergonomic format, with as little corner cases / dangerous corners as possible.

This converts model serialization manifests that record every model file shard hash into an in-toto payload that can then be passed to Sigstore's sign_intoto for signing to generate a Sigstore Bundle (if using Sigstore).

This time, we record every hash as part of the subject instead of in the payload. We require verifiers to be aware of this and acknowledge that verifiers that only check subject by subject (that is, they check if the hash of a passed in argument is in the list of subjects and don't check if all the hashes are present), can fail to fully detect if the model integrity is compromised by renaming one file in the model, interchanging two file names, deleting a file, or reordering two shards. The signing library will have additional checks for this, but verifying the signature with other tools might result in invalid results.

CC @susperius for converting manifest to in-toto. This should cover #111, #224, and #248 (first part of the machinery). CC @laurentsimon and (optionally) @TomHennen to make sure I did not mishandle in-toto.

Note: I still had to pass some payload to the in-toto predicate due to in-toto/attestation#374. Right now, it is a key-val pair that should be ignored, but there is also the possibility of registering only one predicate type for all model signing in-toto formats and then registering a subtype as part of the predicate, that we can control as much as we need.

Note: This is the equivalent of #264, but with digests added as subjects.

Note: This is the equivalent of #266, but for file shard hashes instead of file hashes.

Note: This builds on #266. I decided to split every feature into its own PR to make it easier to review what changes (should be only the last commit) and to be able to merge partial work and continue from there.

Finally, merging this will:

• Close Manifest file #111
• Close Manifest file format #224
• Close Support for converting manifests to in-toto statements #248
• Make WIP feat: Use manifest #112 obsolete
• Probably require removing/re-doing WIP adopting manifests and signing classes #260 and Add signing and verification CLI scripts. #240.

Release Note

NONE

Documentation

NONE

The merge-base changed after approval.

[spencerschrock]

Deferring to Laurent's review, I did confirm Mihai's latest force-push was a rebase on top of main.

Note: In the PR description I'm assuming

Note: This is the equivalent of #265, but for file shard hashes instead of file hashes.

Should be

Note: This is the equivalent of #266, but for file shard hashes instead of file hashes.

[mihaimaruseac]

Thank you for the review!

Note: In the PR description I'm assuming

Note: This is the equivalent of #265, but for file shard hashes instead of file hashes.

Should be

Note: This is the equivalent of #266, but for file shard hashes instead of file hashes.

It should have been 2 notes, one describing the diff from #264 (subjects instead of payload) and one from #266. Fixed now.