Robust file dependency download process

[nichollsh]

Sometimes the tests fail because the model is not able to get data from OSF. This can usually be fixed by requesting the test to be run again, in which case the data usually downloads fine.

Example of it failing: https://github.com/FormingWorlds/PROTEUS/actions/runs/11366750494/job/31617828810

We could include a loop in the downloader functions that re-attempts the download if an error is thrown.

[timlichtenberg]

Issue splits into two issues

• A) Download sometimes fails completely and throws an exception because of server-side issues.
• B) Sometimes it just doesn't download everything, and just a few files or one file is missing. Need some sort of registry or file inventory.

First issue is probably easier to solve than second one.

[nichollsh]

Regarding issue B, we could potentially query OSF for the files we expect to download. This can be done fairly easily with curl.

For example, the node ID for the spectral files is vehxg, so we can obtain a list of all the files with:
curl -X GET https://api.osf.io/v2/nodes/vehxg/files/osfstorage/

This provides a JSON dataset which lists all of the folders and files within the node. We can the recursively follow the folders to make a tree of files. For each file, OSF also provides an MD5 hash that can be used to verify that each of the files downloaded fine.

In the same JSON dataset, we also get download links for each file. We could also consider using this as an alternative to the current osfclient library which hasn't been updated in over 4 years.

[nichollsh]

Working example in the notebook here:
request.ipynb.zip

This provides functions for listing all files in a node, getting the download URL for a given file path on OSF, and obtaining MD5 hashes. This is done without using the old osfclient library.