feat: hash checksums + download urls for snapshot restore and create

[BrendanCoughlan5]

hash checksums + download urls for snapshot restore and create, also cleaned up/ logs/ readme/ docs

[seanmcgary]

Selected the wrong option on accident...see above feedback for changes needed.

[seanmcgary]

Ran the restore-snapshot function and encountered a strange error. It seems to have downloaded the file completely, but then say's it received a 404 after the fact (which isnt possible):

downloading tmp/sidecar-testnet-holesky_v3.0.0-rc.1_public_20250122.dump 100% [========================================] Error: failed to restore snapshot: Error downloading snapshot from 'https://eigenlayer-sidecar.s3.us-east-1.amazonaws.com/snapshots/testnet-holesky/sidecar-testnet-holesky_v3.0.0-rc.1_public_20250122.dump': downloading error, received status code 404

It also didnt clean up the file that was created, which it probably should.

arguments:

go run main.go restore-snapshot
    --ethereum.rpc-url="https://winter-white-crater.ethereum-holesky.quiknode.pro/1b1d75c4ada73b7ad98e1488880649d4ea637733/" \
    --chain="preprod" \
    --database.host="localhost" \
    --database.port="5432" \
    --database.user="seanmcgary" \
    --database.password="" \
    --database.db_name="sidecar_test_restore" \
        --ethereum.use_native_batch_call=false \
        --snapshot.input=https://eigenlayer-sidecar.s3.us-east-1.amazonaws.com/snapshots/testnet-holesky/sidecar-testnet-holesky_v3.0.0-rc.1_public_20250122.dump \
        --database.schema_name=public

As a sidenote, when we encounter an error during execution, can we change it so that the usage message isnt printed? It adds a lot of noise and should really only be printed if the inputs are incorrect.

[BrendanCoughlan5]

Ran the restore-snapshot function and encountered a strange error. It seems to have downloaded the file completely, but then say's it received a 404 after the fact (which isnt possible):

downloading tmp/sidecar-testnet-holesky_v3.0.0-rc.1_public_20250122.dump 100% [========================================] Error: failed to restore snapshot: Error downloading snapshot from 'https://eigenlayer-sidecar.s3.us-east-1.amazonaws.com/snapshots/testnet-holesky/sidecar-testnet-holesky_v3.0.0-rc.1_public_20250122.dump': downloading error, received status code 404

It also didnt clean up the file that was created, which it probably should.

arguments:

go run main.go restore-snapshot
    --ethereum.rpc-url="https://winter-white-crater.ethereum-holesky.quiknode.pro/1b1d75c4ada73b7ad98e1488880649d4ea637733/" \
    --chain="preprod" \
    --database.host="localhost" \
    --database.port="5432" \
    --database.user="seanmcgary" \
    --database.password="" \
    --database.db_name="sidecar_test_restore" \
        --ethereum.use_native_batch_call=false \
        --snapshot.input=https://eigenlayer-sidecar.s3.us-east-1.amazonaws.com/snapshots/testnet-holesky/sidecar-testnet-holesky_v3.0.0-rc.1_public_20250122.dump \
        --database.schema_name=public

As a sidenote, when we encounter an error during execution, can we change it so that the usage message isnt printed? It adds a lot of noise and should really only be printed if the inputs are incorrect.

Fixed to be clear it's the filesum the hash that doesn't exist.

Try with the --snapshot.verify-input=false flag

It should now give something like : {"level":"fatal","ts":"2025-02-03T12:53:34.148-0800","caller":"cmd/restoreSnapshot.go:44","msg":"failed to restore snapshot","error":"snapshot hash file not found at 'https://eigenlayer-sidecar.s3.us-east-1.amazonaws.com/snapshots/testnet-holesky/sidecar-testnet-holesky_v3.0.0-rc.1_public_20250122.dump.sha256sum'. Ensure the file exists or set --verify-input=false to skip verification","stacktrace":"github.com/Layr-Labs/sidecar/cmd.init.func4\n\t/Users/brendan/Documents/GitHub/sidecar/cmd/restoreSnapshot.go:44\ngithub.com/spf13/cobra.(*Command).execute\n\t/Users/brendan/go/pkg/mod/github.com/spf13/[email protected]/command.go:985\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/Users/brendan/go/pkg/mod/github.com/spf13/[email protected]/command.go:1117\ngithub.com/spf13/cobra.(*Command).Execute\n\t/Users/brendan/go/pkg/mod/github.com/spf13/[email protected]/command.go:1041\ngithub.com/Layr-Labs/sidecar/cmd.Execute\n\t/Users/brendan/Documents/GitHub/sidecar/cmd/root.go:19\nmain.main\n\t/Users/brendan/Documents/GitHub/sidecar/main.go:8\nruntime.main\n\t/opt/homebrew/Cellar/go/1.23.4/libexec/src/runtime/proc.go:272"}

[BrendanCoughlan5]

#201 (comment)

downloadFile is intended to be used to download an individual file, like you downloadFile(snapshot), then downloadFile(snapshotHash)

[seanmcgary]

Doesnt look like thats the case. Still calling urlExists which executes a HEAD request

[seanmcgary]

Still making a HEAD request