refactor: sdk v2 #1875
Merged
refactor: sdk v2 #1875
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
chore: merge upstream
chore: merge upstream
chore: merge upstream
chore: merge upstream
chore: upstream
chore: bump `dev`
chore: merge upstream
chore: update
* working with the 'pass-through' method * cleanup * making clippy happy
jtguibas
reviewed
Dec 18, 2024
jtguibas
approved these changes
Dec 19, 2024
ctian1
added a commit
that referenced
this pull request
Dec 25, 2024
* fix workflows * fix * remove fixtures * cleanup * fix * fix * assertions * trace * add is_memory to instruction interactions * a * alu_base * f * preprocessed batch_fri * preprocessed batch_fri * preprocessed * gen trace * fri_fold * fri-fold * public_values * public_values * preprocessed * gen_trace * skinny * poseidon2_skinny * poseidon2_wide * switch * assert * assert_eq * feat: allow for big shapes (#98) * new partial ord * test * maximal set * script v1 * some iterator methods * shape json * maximal shapes * more shapes * program list * shape script with default set * refactor local counts * update syscall counts * change position * local mem counts * fix shape count and distance * fix test * maximal core shapes * weak ordering * cleanup * by shard * checkpoint * shapes * reset option * try * refactor * checkpoint * checkpoint * checkpoint * fix script * back script * more mem init * one ordering only * all programs * checkpoint * checkpoint * try * cleanup * comment * checkpoint * move shape script * clean script * checkpoint try to clean * move script back * private * cleanup * fix cli * script * cleanup * tune * new shapes * fix * threshold * clear only shard sizes given * bound * fix counts * limited shapes * bound * pub * shapes * typo * test * wr too * wr * max shapes * try * f * fix counts * new shapes * 21 shaeps * fix checkpoint * checkpoint * new shapes * go * fix bounds * ff * shapes * fix max shape * counts * checkpoint * checkpoint * checkpoint * update * clippy * try * checkpoint * hacky * f * rm * span * merge latest changes * hm weird --------- Co-authored-by: jtguibas <[email protected]> * handle op a set to register 0 correctly * gpu tracegen fix * fix * feat(recursion): executor parallelization (#66) * change cycle tracker to use `Cow<'static, str>` * remove vestigial cycle tracker * basic data structures for the new program * new program struct in runtime, iterator impls * replace program assoc type in airs * add cycle tracker recursion macro * fix cycle tracker span double counting (already fixed in sp1) * sub builder helper method * add parallel builder to recursion * refactor bounds in recursion compiler * parallel dsl support in compiler * replace old compile function * legacy program behind cfg(test) * unsafecell runtime attempt 2 * attempt 2 to separate execute_one out * execute_raw * clean up `MemVec` slightly * witness stream accessible only at top level * parallelize another loop * safety comment for recursion run * better tracing spans in compiler * safety newtype * more safety comments in recursion runtime * clean up a bit * strip out Option in MemVec * `MaybeUninit` in memory * metadata field in program * fixed/upgraded `print_stats` * (doesn't work) validation * copied smallvec for recursion validation * safety wrapper for `DslIrBlock`, undocumented * safety comments * fix constraints to work with parallel instruction * validate memory * re-implement debug printing * fix recursion tests * remove old `RecursionProgram` struct * inline `ProgramMetadata`, have only `shape_mut` on `RecProgram` * fix spelling in validation error * rename `RecProgram` back to `RecursionProgram` * clean up `memory.rs` * revert printing stats in compress * revert changes to root `Cargo.toml` * clippy * don't juggle in hot loop, init alpha_pows all at once * fix: don't push `new_alpha` again * add more spans and optimize hint reading * remove intermediate collects (perf untested) * fix soundness in mw_unchecked * fix memory for real, add safety comments * fix typos * fix a safety comment for RecursionProgram * fix bounds for `SyncUnsafeCell` * fix recursion chip tests * remove instruction extractors * run `cargo clippy --fix`, fix minor lints * automatically set `total_memory` in tests * type annotate `transmute`s, run `cargo fmt` * change `.cloned()` into `.copied()` in `io_addrs` * disable failing tests (to be fixed later) --------- Co-authored-by: tamirhemo <[email protected]> Co-authored-by: jtguibas <[email protected]> * verify next_pc for non halt ecall * column counts changed * re-enable shapes for unit tests * made program mutable * updates * better tuning for gpu * in progress shape cleanup * succesful shape migration * more cleanup * cleanup * collection checkpoint * built shapes * hm * artifacts * remove panic handle * fix small bugs * fix small issues * update * more update * adjust executor logic * tiny shapes * cleanup * small shapes * fix some small errors * add missing file * verified that address is > 0x20 * Update crates/core/machine/src/memory/instructions/air.rs Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix: shape iterator * fixed formatting issue * use staging rpc * update version to rc3 * perf * feat: tests + sp1-perf for executor (#118) * perf: no nonces in executor (#119) * perf: register file (#120) * cumulative sum bug fix * fixed core_air_cost_consistency * register 32 is a valid mem addr * compile flag `c++20` -> c++17` * removed four permutation columns * check that memory instr don't access registers in executor and fixed column cnt file * fixed executor addr check * updated deviations doc * test * fix some issues * fixed conflict * hm * workflow * separate out executor and prover stuff * add executor-suite * wrong command in executor-suite * install m4 in ci * clean up prover opts * cleanup sp1 split opts * opts * fixed comment * in progress changes * fix * hm * fix execution stopping * latest changes * groth16 * name (#131) * remove nonce from the u256xu2048 table * cleanup * latest changes * changes * cpu param selection * adjust heuristic * adjust parameters * auto detect avx512 * update opts * update * opts * boom * commit maximal shapes * tune params * changes * hm * check memory writes * more changes * adjustments * foom * changes * changes * prevent cpu oom on reth * add blacklisted shapes * hm * update cuda image * more worst case maximal shapes * executor * cleanup generate deps span * get rid of prints * fix attempt * improve comments * fix vk hash * additional protection on is_complete * simplify AIR, add comment * mini cleanup, comments * tune number for 4xlarge * comments and very minor changes * Update crates/core/machine/src/control_flow/auipc.rs Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * shape selection fix * remove global 20 shapes from 2^19 * remove global 22 shapes from cpu 21 * push missing file * new recursion shape * update si * more cleanup * more small changes * new shape finding logic * hm * check memory reads * revert some changes * optimistic commit * some reverts * fix rv32im costs * update pr to work * fix: shape panic handling * fix: handle panic during shape vkey setup * vk map * fix * impact * push * recursion shape * do * fix script * generate back * test * print panic shapes * fix recursion script * make shape bigger * shape update * beq failure test * cleaner types * fix * verify vk true * changes * addresses PR comment * test shrink * some comment fixes * latest commit * small shapes checkpoint * new vk map * vk map * Revert "task(core): malicious trace generator support for failure test cases" * update action * update action * update shapes * new shapes * perf: executor performance (#145) * which events are exceeding * coreairid * cleanup -> 37 on mac * more cleanup * fix the test * get script to compile ; * reorder airs * change default to auto * comment nit * Revert "perf: executor performance (#145)" This reverts commit 7344448. * fix: executor oom * perf: executor performance (#145) * which events are exceeding * coreairid * cleanup -> 37 on mac * more cleanup * fix the test * get script to compile ; * reorder airs * change default to auto * comment nit * Revert "perf: executor performance (#145)" This reverts commit af0b64e. * Reapply "perf: executor performance (#145)" This reverts commit ae08568. * fix executor commit * fix * disable `sp1-recursion-compiler/debug` feature by default (#154) * feat(recursion): print backtraces on failed assertions (#155) * replace `TracedVec` with `Vec` * add `DebugBacktrace` to dsl and executor instructions * print backtraces with feature `"debug"` on division by zero * add `--no-fail-fast` to CI. should revert when CI passes * update gitignore * clippy fix * fix clippy * update release * update vk * more cleanup * Update crates/core/executor/src/profiler.rs Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix issues * fix: zkvm workspace version (#166) * fix ssz withdrawals (#165) * refactor: sdk v2 (#1875) Co-authored-by: Tamir Hemo <[email protected]> Co-authored-by: Tej Qu Nair <[email protected]> Co-authored-by: Kevin Jue <[email protected]> Co-authored-by: John Guibas <[email protected]> Co-authored-by: erabinov <[email protected]> Co-authored-by: Eugene Rabinovich <[email protected]> Co-authored-by: Yuwen Zhang <[email protected]> Co-authored-by: N <[email protected]> * fix format * fix examples * fix executor clippy * changes * modify * fix sdk * cleanup files a bit * update groth16_vk * fix shape better algo * shape fix * fix shape matching logic * fix fix_shape * new vk map --------- Co-authored-by: Gautham Elango <[email protected]> Co-authored-by: Kevin Jue <[email protected]> Co-authored-by: Gautham Elango <[email protected]> Co-authored-by: Tamir Hemo <[email protected]> Co-authored-by: Tej Qu Nair <[email protected]> Co-authored-by: tamirhemo <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Chris Tian <[email protected]> Co-authored-by: Bhargav Annem <[email protected]> Co-authored-by: Yuwen Zhang <[email protected]> Co-authored-by: rkm0959 <[email protected]> Co-authored-by: erabinov <[email protected]> Co-authored-by: Eugene Rabinovich <[email protected]> Co-authored-by: N <[email protected]>
ctian1
added a commit
that referenced
this pull request
Dec 25, 2024
* feat: allow for big shapes (#98) * new partial ord * test * maximal set * script v1 * some iterator methods * shape json * maximal shapes * more shapes * program list * shape script with default set * refactor local counts * update syscall counts * change position * local mem counts * fix shape count and distance * fix test * maximal core shapes * weak ordering * cleanup * by shard * checkpoint * shapes * reset option * try * refactor * checkpoint * checkpoint * checkpoint * fix script * back script * more mem init * one ordering only * all programs * checkpoint * checkpoint * try * cleanup * comment * checkpoint * move shape script * clean script * checkpoint try to clean * move script back * private * cleanup * fix cli * script * cleanup * tune * new shapes * fix * threshold * clear only shard sizes given * bound * fix counts * limited shapes * bound * pub * shapes * typo * test * wr too * wr * max shapes * try * f * fix counts * new shapes * 21 shaeps * fix checkpoint * checkpoint * new shapes * go * fix bounds * ff * shapes * fix max shape * counts * checkpoint * checkpoint * checkpoint * update * clippy * try * checkpoint * hacky * f * rm * span * merge latest changes * hm weird --------- Co-authored-by: jtguibas <[email protected]> * handle op a set to register 0 correctly * gpu tracegen fix * fix * feat(recursion): executor parallelization (#66) * change cycle tracker to use `Cow<'static, str>` * remove vestigial cycle tracker * basic data structures for the new program * new program struct in runtime, iterator impls * replace program assoc type in airs * add cycle tracker recursion macro * fix cycle tracker span double counting (already fixed in sp1) * sub builder helper method * add parallel builder to recursion * refactor bounds in recursion compiler * parallel dsl support in compiler * replace old compile function * legacy program behind cfg(test) * unsafecell runtime attempt 2 * attempt 2 to separate execute_one out * execute_raw * clean up `MemVec` slightly * witness stream accessible only at top level * parallelize another loop * safety comment for recursion run * better tracing spans in compiler * safety newtype * more safety comments in recursion runtime * clean up a bit * strip out Option in MemVec * `MaybeUninit` in memory * metadata field in program * fixed/upgraded `print_stats` * (doesn't work) validation * copied smallvec for recursion validation * safety wrapper for `DslIrBlock`, undocumented * safety comments * fix constraints to work with parallel instruction * validate memory * re-implement debug printing * fix recursion tests * remove old `RecursionProgram` struct * inline `ProgramMetadata`, have only `shape_mut` on `RecProgram` * fix spelling in validation error * rename `RecProgram` back to `RecursionProgram` * clean up `memory.rs` * revert printing stats in compress * revert changes to root `Cargo.toml` * clippy * don't juggle in hot loop, init alpha_pows all at once * fix: don't push `new_alpha` again * add more spans and optimize hint reading * remove intermediate collects (perf untested) * fix soundness in mw_unchecked * fix memory for real, add safety comments * fix typos * fix a safety comment for RecursionProgram * fix bounds for `SyncUnsafeCell` * fix recursion chip tests * remove instruction extractors * run `cargo clippy --fix`, fix minor lints * automatically set `total_memory` in tests * type annotate `transmute`s, run `cargo fmt` * change `.cloned()` into `.copied()` in `io_addrs` * disable failing tests (to be fixed later) --------- Co-authored-by: tamirhemo <[email protected]> Co-authored-by: jtguibas <[email protected]> * verify next_pc for non halt ecall * column counts changed * re-enable shapes for unit tests * made program mutable * updates * better tuning for gpu * in progress shape cleanup * succesful shape migration * more cleanup * cleanup * collection checkpoint * built shapes * hm * artifacts * remove panic handle * fix small bugs * fix small issues * update * more update * adjust executor logic * tiny shapes * cleanup * small shapes * fix some small errors * add missing file * verified that address is > 0x20 * Update crates/core/machine/src/memory/instructions/air.rs Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix: shape iterator * fixed formatting issue * use staging rpc * update version to rc3 * perf * feat: tests + sp1-perf for executor (#118) * perf: no nonces in executor (#119) * perf: register file (#120) * cumulative sum bug fix * fixed core_air_cost_consistency * register 32 is a valid mem addr * compile flag `c++20` -> c++17` * removed four permutation columns * check that memory instr don't access registers in executor and fixed column cnt file * fixed executor addr check * updated deviations doc * test * fix some issues * fixed conflict * hm * workflow * separate out executor and prover stuff * add executor-suite * wrong command in executor-suite * install m4 in ci * clean up prover opts * cleanup sp1 split opts * opts * fixed comment * in progress changes * fix * hm * fix execution stopping * latest changes * groth16 * name (#131) * remove nonce from the u256xu2048 table * cleanup * latest changes * changes * cpu param selection * adjust heuristic * adjust parameters * auto detect avx512 * update opts * update * opts * boom * commit maximal shapes * tune params * changes * hm * check memory writes * more changes * adjustments * foom * changes * changes * prevent cpu oom on reth * add blacklisted shapes * hm * update cuda image * more worst case maximal shapes * executor * cleanup generate deps span * get rid of prints * fix attempt * improve comments * fix vk hash * additional protection on is_complete * simplify AIR, add comment * mini cleanup, comments * tune number for 4xlarge * comments and very minor changes * Update crates/core/machine/src/control_flow/auipc.rs Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * shape selection fix * remove global 20 shapes from 2^19 * remove global 22 shapes from cpu 21 * push missing file * new recursion shape * update si * more cleanup * more small changes * new shape finding logic * hm * check memory reads * revert some changes * optimistic commit * some reverts * fix rv32im costs * update pr to work * fix: shape panic handling * fix: handle panic during shape vkey setup * vk map * fix * impact * push * recursion shape * do * fix script * generate back * test * print panic shapes * fix recursion script * make shape bigger * shape update * beq failure test * cleaner types * fix * verify vk true * changes * addresses PR comment * test shrink * some comment fixes * latest commit * small shapes checkpoint * new vk map * vk map * Revert "task(core): malicious trace generator support for failure test cases" * update action * update action * update shapes * new shapes * perf: executor performance (#145) * which events are exceeding * coreairid * cleanup -> 37 on mac * more cleanup * fix the test * get script to compile ; * reorder airs * change default to auto * comment nit * Revert "perf: executor performance (#145)" This reverts commit 7344448. * fix: executor oom * perf: executor performance (#145) * which events are exceeding * coreairid * cleanup -> 37 on mac * more cleanup * fix the test * get script to compile ; * reorder airs * change default to auto * comment nit * Revert "perf: executor performance (#145)" This reverts commit af0b64e. * Reapply "perf: executor performance (#145)" This reverts commit ae08568. * fix executor commit * fix * disable `sp1-recursion-compiler/debug` feature by default (#154) * feat(recursion): print backtraces on failed assertions (#155) * replace `TracedVec` with `Vec` * add `DebugBacktrace` to dsl and executor instructions * print backtraces with feature `"debug"` on division by zero * add `--no-fail-fast` to CI. should revert when CI passes * update gitignore * clippy fix * fix clippy * update release * update vk * generate proofs in tests * nit * more cleanup * Update crates/core/executor/src/profiler.rs Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix issues * fix: zkvm workspace version (#166) * fix ssz withdrawals (#165) * refactor: sdk v2 (#1875) Co-authored-by: Tamir Hemo <[email protected]> Co-authored-by: Tej Qu Nair <[email protected]> Co-authored-by: Kevin Jue <[email protected]> Co-authored-by: John Guibas <[email protected]> Co-authored-by: erabinov <[email protected]> Co-authored-by: Eugene Rabinovich <[email protected]> Co-authored-by: Yuwen Zhang <[email protected]> Co-authored-by: N <[email protected]> * separate out veriifer crate test * fix format * fix examples * fix executor clippy * changes * modify * fix sdk * cleanup files a bit * update groth16_vk * fix shape better algo * shape fix * fix shape matching logic * fix fix_shape * new vk map * new sdk in verifier crate * feat(sdk): `get_proof_status`, `request`, `cycle_limit` (#1883) * feat(sdk): latest network proto + error handling + fix examples (#1884) * feat: use `B256` + `request_async` + bump `alloy` + fix CI (#1886) * docs: add conventional commit guidelines to CONTRIBUTING.md (#1854) * fix: remove stdin from SP1ProofWithPublicValues (#1714) * fix: default tag * feat: update book, imports * fix * use SP1_CIRCUIT_VERSION * docs * fix * fix * feat: change loc, fix later * add * add * add * fix * update vk * fix examples * update vk * lock * fmt --------- Co-authored-by: Kevin Jue <[email protected]> Co-authored-by: Tamir Hemo <[email protected]> Co-authored-by: Tej Qu Nair <[email protected]> Co-authored-by: tamirhemo <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Chris Tian <[email protected]> Co-authored-by: Bhargav Annem <[email protected]> Co-authored-by: Yuwen Zhang <[email protected]> Co-authored-by: rkm0959 <[email protected]> Co-authored-by: erabinov <[email protected]> Co-authored-by: Eugene Rabinovich <[email protected]> Co-authored-by: N <[email protected]> Co-authored-by: Ratan Kaliani <[email protected]> Co-authored-by: Matt Stam <[email protected]> Co-authored-by: Jev Björsell <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Solution
PR Checklist