Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(sdk): latest network proto + error handling + fix examples #1884

Merged
merged 13 commits into from
Dec 20, 2024

Conversation

mattstam
Copy link
Member

@mattstam mattstam commented Dec 20, 2024

This PR updates the network SDK with:

  • Updated proto
  • Custom error types using thiserror (e.g. to implement different logic for unexecutable vs unfulfillable)
  • Removal all unwrap / expect statements and bubble up all errors correctly
  • Fix all examples / docs to borrow the stdin

Error handling example:

    // Handle errors from proof generation.
    let mut proof = match proof_result {
        Ok(proof) => proof,
        Err(e) => {
            if let Some(network_error) = e.downcast_ref::<Error>() {
                match network_error {
                    Error::RequestUnexecutable { request_id: _ } => {
                        eprintln!("Program is unexecutable: {}", e);
                        std::process::exit(1);
                    }
                    Error::RequestUnfulfillable { request_id: _ } => {
                        eprintln!("Proof request cannot be fulfilled: {}", e);
                        std::process::exit(1);
                    }
                    _ => {
                        eprintln!("Unexpected error: {}", e);
                        std::process::exit(1);
                    }
                }
            } else {
                eprintln!("Unexpected error: {}", e);
                std::process::exit(1);
            }
        }
    };

Example of error bubbling up and handling it:

2024-12-20T20:37:57.724135Z  INFO execute: clk = 0 pc = 0x2017c0    
2024-12-20T20:37:57.735063Z  INFO execute: close time.busy=15.2ms time.idle=35.8µs
executed program with 13408 cycles
2024-12-20T20:38:01.951775Z  INFO Registered program 0x00cd8955b7884b1d70dfec2223eb86b3fa240f288582f9924ab7cdf680be95c8    
2024-12-20T20:38:02.025432Z  INFO execute: clk = 0 pc = 0x2017c0    
2024-12-20T20:38:02.037855Z  INFO execute: close time.busy=86.1ms time.idle=2.00µs
2024-12-20T20:38:02.037890Z  INFO Requesting proof:    
2024-12-20T20:38:02.037893Z  INFO ├─ Cycle limit: 13408    
2024-12-20T20:38:02.037895Z  INFO ├─ Proof mode: Core    
2024-12-20T20:38:02.037896Z  INFO ├─ Strategy: Hosted    
2024-12-20T20:38:02.037897Z  INFO ├─ Timeout: 14400 seconds    
2024-12-20T20:38:02.037899Z  INFO └─ Circuit version: v3.0.0    
2024-12-20T20:38:03.885853Z  INFO Created request 0x9d805d27cb1c77431f2fcaa1737e5647add2a5099378bed37adaf70fc791ee22 in transaction 0x3e4f4db8b8d75f54c97991461aefd3d16cb392a5d39fee04cb68866ed7e0e801    
2024-12-20T20:38:04.322782Z  INFO Proof request assigned, proving...    
Program is unexecutable: Proof request 0x9d805d27cb1c77431f2fcaa1737e5647add2a5099378bed37adaf70fc791ee22 is unexecutable

In a real situation, you may want to retry if it's unfulfillable or implement custom logic.

@mattstam mattstam changed the title feat: update SDK for network feat(sdk): latest network proto + error handling Dec 20, 2024
Copy link
Contributor

github-actions bot commented Dec 20, 2024

SP1 Performance Test Results

Branch: mattstam/sdk-fix
Commit: 175acd2
Author: mattstam

program cycles execute (mHz) core (kHZ) compress (KHz) time success
fibonacci 11291 0.17 2.79 0.45 25s
ssz-withdrawals 2757356 16.46 128.58 38.55 1m12s
tendermint 12593597 6.69 264.28 104.40 2m3s

@mattstam mattstam changed the title feat(sdk): latest network proto + error handling feat(sdk): latest network proto + error handling + fix examples Dec 20, 2024
Copy link
Member

@ratankaliani ratankaliani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mattstam mattstam merged commit a1e3623 into dev Dec 20, 2024
13 of 16 checks passed
@mattstam mattstam deleted the mattstam/sdk-fix branch December 20, 2024 22:03
ctian1 added a commit that referenced this pull request Dec 25, 2024
* feat: allow for big shapes (#98)

* new partial ord

* test

* maximal set

* script v1

* some iterator methods

* shape json

* maximal shapes

* more shapes

* program list

* shape script with default set

* refactor local counts

* update syscall counts

* change position

* local mem counts

* fix shape count and distance

* fix test

* maximal core shapes

* weak ordering

* cleanup

* by shard

* checkpoint

* shapes

* reset option

* try

* refactor

* checkpoint

* checkpoint

* checkpoint

* fix script

* back script

* more mem init

* one ordering only

* all programs

* checkpoint

* checkpoint

* try

* cleanup

* comment

* checkpoint

* move shape script

* clean script

* checkpoint try to clean

* move script back

* private

* cleanup

* fix cli

* script

* cleanup

* tune

* new shapes

* fix

* threshold

* clear only shard sizes given

* bound

* fix counts

* limited shapes

* bound

* pub

* shapes

* typo

* test

* wr too

* wr

* max shapes

* try

* f

* fix counts

* new shapes

* 21 shaeps

* fix checkpoint

* checkpoint

* new shapes

* go

* fix bounds

* ff

* shapes

* fix max shape

* counts

* checkpoint

* checkpoint

* checkpoint

* update

* clippy

* try

* checkpoint

* hacky

* f

* rm

* span

* merge latest changes

* hm weird

---------

Co-authored-by: jtguibas <[email protected]>

* handle op a set to register 0 correctly

* gpu tracegen fix

* fix

* feat(recursion): executor parallelization (#66)

* change cycle tracker to use `Cow<'static, str>`

* remove vestigial cycle tracker

* basic data structures for the new program

* new program struct in runtime, iterator impls

* replace program assoc type in airs

* add cycle tracker recursion macro

* fix cycle tracker span double counting (already fixed in sp1)

* sub builder helper method

* add parallel builder to recursion

* refactor bounds in recursion compiler

* parallel dsl support in compiler

* replace old compile function

* legacy program behind cfg(test)

* unsafecell runtime attempt 2

* attempt 2 to separate execute_one out

* execute_raw

* clean up `MemVec` slightly

* witness stream accessible only at top level

* parallelize another loop

* safety comment for recursion run

* better tracing spans in compiler

* safety newtype

* more safety comments in recursion runtime

* clean up a bit

* strip out Option in MemVec

* `MaybeUninit` in memory

* metadata field in program

* fixed/upgraded `print_stats`

* (doesn't work) validation

* copied smallvec for recursion validation

* safety wrapper for `DslIrBlock`, undocumented

* safety comments

* fix constraints to work with parallel instruction

* validate memory

* re-implement debug printing

* fix recursion tests

* remove old `RecursionProgram` struct

* inline `ProgramMetadata`, have only `shape_mut` on `RecProgram`

* fix spelling in validation error

* rename `RecProgram` back to `RecursionProgram`

* clean up `memory.rs`

* revert printing stats in compress

* revert changes to root `Cargo.toml`

* clippy

* don't juggle in hot loop, init alpha_pows all at once

* fix: don't push `new_alpha` again

* add more spans and optimize hint reading

* remove intermediate collects (perf untested)

* fix soundness in mw_unchecked

* fix memory for real, add safety comments

* fix typos

* fix a safety comment for RecursionProgram

* fix bounds for `SyncUnsafeCell`

* fix recursion chip tests

* remove instruction extractors

* run `cargo clippy --fix`, fix minor lints

* automatically set `total_memory` in tests

* type annotate `transmute`s, run `cargo fmt`

* change `.cloned()` into `.copied()` in `io_addrs`

* disable failing tests (to be fixed later)

---------

Co-authored-by: tamirhemo <[email protected]>
Co-authored-by: jtguibas <[email protected]>

* verify next_pc for non halt ecall

* column counts changed

* re-enable shapes for unit tests

* made program mutable

* updates

* better tuning for gpu

* in progress shape cleanup

* succesful shape migration

* more cleanup

* cleanup

* collection checkpoint

* built shapes

* hm

* artifacts

* remove panic handle

* fix small bugs

* fix small issues

* update

* more update

* adjust executor logic

* tiny shapes

* cleanup

* small shapes

* fix some small errors

* add missing file

* verified that address is > 0x20

* Update crates/core/machine/src/memory/instructions/air.rs

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* fix: shape iterator

* fixed formatting issue

* use staging rpc

* update version to rc3

* perf

* feat: tests + sp1-perf for executor (#118)

* perf: no nonces in executor (#119)

* perf: register file (#120)

* cumulative sum bug fix

* fixed core_air_cost_consistency

* register 32 is a valid mem addr

* compile flag `c++20` -> c++17`

* removed four permutation columns

* check that memory instr don't access registers in executor and fixed column cnt file

* fixed executor addr check

* updated deviations doc

* test

* fix some issues

* fixed conflict

* hm

* workflow

* separate out executor and prover stuff

* add executor-suite

* wrong command in executor-suite

* install m4 in ci

* clean up prover opts

* cleanup sp1 split opts

* opts

* fixed comment

* in progress changes

* fix

* hm

* fix execution stopping

* latest changes

* groth16

* name (#131)

* remove nonce from the u256xu2048 table

* cleanup

* latest changes

* changes

* cpu param selection

* adjust heuristic

* adjust parameters

* auto detect avx512

* update opts

* update

* opts

* boom

* commit maximal shapes

* tune params

* changes

* hm

* check memory writes

* more changes

* adjustments

* foom

* changes

* changes

* prevent cpu oom on reth

* add blacklisted shapes

* hm

* update cuda image

* more worst case maximal shapes

* executor

* cleanup generate deps span

* get rid of prints

* fix attempt

* improve comments

* fix vk hash

* additional protection on is_complete

* simplify AIR, add comment

* mini cleanup, comments

* tune number for 4xlarge

* comments and very minor changes

* Update crates/core/machine/src/control_flow/auipc.rs

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* shape selection fix

* remove global 20 shapes from 2^19

* remove global 22 shapes from cpu 21

* push missing file

* new recursion shape

* update si

* more cleanup

* more small changes

* new shape finding logic

* hm

* check memory reads

* revert some changes

* optimistic commit

* some reverts

* fix rv32im costs

* update pr to work

* fix: shape panic handling

* fix: handle panic during shape vkey setup

* vk map

* fix

* impact

* push

* recursion shape

* do

* fix script

* generate back

* test

* print panic shapes

* fix recursion script

* make shape bigger

* shape update

* beq failure test

* cleaner types

* fix

* verify vk true

* changes

* addresses PR comment

* test shrink

* some comment fixes

* latest commit

* small shapes checkpoint

* new vk map

* vk map

* Revert "task(core): malicious trace generator support for failure test cases"

* update action

* update action

* update shapes

* new shapes

* perf: executor performance (#145)

* which events are exceeding

* coreairid

* cleanup -> 37 on mac

* more cleanup

* fix the test

* get script to compile
;

* reorder airs

* change default to auto

* comment nit

* Revert "perf: executor performance (#145)"

This reverts commit 7344448.

* fix: executor oom

* perf: executor performance (#145)

* which events are exceeding

* coreairid

* cleanup -> 37 on mac

* more cleanup

* fix the test

* get script to compile
;

* reorder airs

* change default to auto

* comment nit

* Revert "perf: executor performance (#145)"

This reverts commit af0b64e.

* Reapply "perf: executor performance (#145)"

This reverts commit ae08568.

* fix executor commit

* fix

* disable `sp1-recursion-compiler/debug` feature by default (#154)

* feat(recursion): print backtraces on failed assertions (#155)

* replace `TracedVec` with `Vec`

* add `DebugBacktrace` to dsl and executor instructions

* print backtraces with feature `"debug"` on division by zero

* add `--no-fail-fast` to CI. should revert when CI passes

* update gitignore

* clippy fix

* fix clippy

* update release

* update vk

* generate proofs in tests

* nit

* more cleanup

* Update crates/core/executor/src/profiler.rs

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* fix issues

* fix: zkvm workspace version (#166)

* fix ssz withdrawals (#165)

* refactor: sdk v2 (#1875)

Co-authored-by: Tamir Hemo <[email protected]>
Co-authored-by: Tej Qu Nair <[email protected]>
Co-authored-by: Kevin Jue <[email protected]>
Co-authored-by: John Guibas <[email protected]>
Co-authored-by: erabinov <[email protected]>
Co-authored-by: Eugene Rabinovich <[email protected]>
Co-authored-by: Yuwen Zhang <[email protected]>
Co-authored-by: N <[email protected]>

* separate out veriifer crate test

* fix format

* fix examples

* fix executor clippy

* changes

* modify

* fix sdk

* cleanup files a bit

* update groth16_vk

* fix shape better algo

* shape fix

* fix shape matching logic

* fix fix_shape

* new vk map

* new sdk in verifier crate

* feat(sdk): `get_proof_status`, `request`, `cycle_limit` (#1883)

* feat(sdk): latest network proto + error handling + fix examples (#1884)

* feat: use `B256` + `request_async` + bump `alloy` + fix CI (#1886)

* docs: add conventional commit guidelines to CONTRIBUTING.md (#1854)

* fix: remove stdin from SP1ProofWithPublicValues (#1714)

* fix: default tag

* feat: update book, imports

* fix

* use SP1_CIRCUIT_VERSION

* docs

* fix

* fix

* feat: change loc, fix later

* add

* add

* add

* fix

* update vk

* fix examples

* update vk

* lock

* fmt

---------

Co-authored-by: Kevin Jue <[email protected]>
Co-authored-by: Tamir Hemo <[email protected]>
Co-authored-by: Tej Qu Nair <[email protected]>
Co-authored-by: tamirhemo <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Chris Tian <[email protected]>
Co-authored-by: Bhargav Annem <[email protected]>
Co-authored-by: Yuwen Zhang <[email protected]>
Co-authored-by: rkm0959 <[email protected]>
Co-authored-by: erabinov <[email protected]>
Co-authored-by: Eugene Rabinovich <[email protected]>
Co-authored-by: N <[email protected]>
Co-authored-by: Ratan Kaliani <[email protected]>
Co-authored-by: Matt Stam <[email protected]>
Co-authored-by: Jev Björsell <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants