Skip to content

Commit

Permalink
feat: dev sync (#169)
Browse files Browse the repository at this point in the history
* feat: allow for big shapes (#98)

* new partial ord

* test

* maximal set

* script v1

* some iterator methods

* shape json

* maximal shapes

* more shapes

* program list

* shape script with default set

* refactor local counts

* update syscall counts

* change position

* local mem counts

* fix shape count and distance

* fix test

* maximal core shapes

* weak ordering

* cleanup

* by shard

* checkpoint

* shapes

* reset option

* try

* refactor

* checkpoint

* checkpoint

* checkpoint

* fix script

* back script

* more mem init

* one ordering only

* all programs

* checkpoint

* checkpoint

* try

* cleanup

* comment

* checkpoint

* move shape script

* clean script

* checkpoint try to clean

* move script back

* private

* cleanup

* fix cli

* script

* cleanup

* tune

* new shapes

* fix

* threshold

* clear only shard sizes given

* bound

* fix counts

* limited shapes

* bound

* pub

* shapes

* typo

* test

* wr too

* wr

* max shapes

* try

* f

* fix counts

* new shapes

* 21 shaeps

* fix checkpoint

* checkpoint

* new shapes

* go

* fix bounds

* ff

* shapes

* fix max shape

* counts

* checkpoint

* checkpoint

* checkpoint

* update

* clippy

* try

* checkpoint

* hacky

* f

* rm

* span

* merge latest changes

* hm weird

---------

Co-authored-by: jtguibas <[email protected]>

* handle op a set to register 0 correctly

* gpu tracegen fix

* fix

* feat(recursion): executor parallelization (#66)

* change cycle tracker to use `Cow<'static, str>`

* remove vestigial cycle tracker

* basic data structures for the new program

* new program struct in runtime, iterator impls

* replace program assoc type in airs

* add cycle tracker recursion macro

* fix cycle tracker span double counting (already fixed in sp1)

* sub builder helper method

* add parallel builder to recursion

* refactor bounds in recursion compiler

* parallel dsl support in compiler

* replace old compile function

* legacy program behind cfg(test)

* unsafecell runtime attempt 2

* attempt 2 to separate execute_one out

* execute_raw

* clean up `MemVec` slightly

* witness stream accessible only at top level

* parallelize another loop

* safety comment for recursion run

* better tracing spans in compiler

* safety newtype

* more safety comments in recursion runtime

* clean up a bit

* strip out Option in MemVec

* `MaybeUninit` in memory

* metadata field in program

* fixed/upgraded `print_stats`

* (doesn't work) validation

* copied smallvec for recursion validation

* safety wrapper for `DslIrBlock`, undocumented

* safety comments

* fix constraints to work with parallel instruction

* validate memory

* re-implement debug printing

* fix recursion tests

* remove old `RecursionProgram` struct

* inline `ProgramMetadata`, have only `shape_mut` on `RecProgram`

* fix spelling in validation error

* rename `RecProgram` back to `RecursionProgram`

* clean up `memory.rs`

* revert printing stats in compress

* revert changes to root `Cargo.toml`

* clippy

* don't juggle in hot loop, init alpha_pows all at once

* fix: don't push `new_alpha` again

* add more spans and optimize hint reading

* remove intermediate collects (perf untested)

* fix soundness in mw_unchecked

* fix memory for real, add safety comments

* fix typos

* fix a safety comment for RecursionProgram

* fix bounds for `SyncUnsafeCell`

* fix recursion chip tests

* remove instruction extractors

* run `cargo clippy --fix`, fix minor lints

* automatically set `total_memory` in tests

* type annotate `transmute`s, run `cargo fmt`

* change `.cloned()` into `.copied()` in `io_addrs`

* disable failing tests (to be fixed later)

---------

Co-authored-by: tamirhemo <[email protected]>
Co-authored-by: jtguibas <[email protected]>

* verify next_pc for non halt ecall

* column counts changed

* re-enable shapes for unit tests

* made program mutable

* updates

* better tuning for gpu

* in progress shape cleanup

* succesful shape migration

* more cleanup

* cleanup

* collection checkpoint

* built shapes

* hm

* artifacts

* remove panic handle

* fix small bugs

* fix small issues

* update

* more update

* adjust executor logic

* tiny shapes

* cleanup

* small shapes

* fix some small errors

* add missing file

* verified that address is > 0x20

* Update crates/core/machine/src/memory/instructions/air.rs

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* fix: shape iterator

* fixed formatting issue

* use staging rpc

* update version to rc3

* perf

* feat: tests + sp1-perf for executor (#118)

* perf: no nonces in executor (#119)

* perf: register file (#120)

* cumulative sum bug fix

* fixed core_air_cost_consistency

* register 32 is a valid mem addr

* compile flag `c++20` -> c++17`

* removed four permutation columns

* check that memory instr don't access registers in executor and fixed column cnt file

* fixed executor addr check

* updated deviations doc

* test

* fix some issues

* fixed conflict

* hm

* workflow

* separate out executor and prover stuff

* add executor-suite

* wrong command in executor-suite

* install m4 in ci

* clean up prover opts

* cleanup sp1 split opts

* opts

* fixed comment

* in progress changes

* fix

* hm

* fix execution stopping

* latest changes

* groth16

* name (#131)

* remove nonce from the u256xu2048 table

* cleanup

* latest changes

* changes

* cpu param selection

* adjust heuristic

* adjust parameters

* auto detect avx512

* update opts

* update

* opts

* boom

* commit maximal shapes

* tune params

* changes

* hm

* check memory writes

* more changes

* adjustments

* foom

* changes

* changes

* prevent cpu oom on reth

* add blacklisted shapes

* hm

* update cuda image

* more worst case maximal shapes

* executor

* cleanup generate deps span

* get rid of prints

* fix attempt

* improve comments

* fix vk hash

* additional protection on is_complete

* simplify AIR, add comment

* mini cleanup, comments

* tune number for 4xlarge

* comments and very minor changes

* Update crates/core/machine/src/control_flow/auipc.rs

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* shape selection fix

* remove global 20 shapes from 2^19

* remove global 22 shapes from cpu 21

* push missing file

* new recursion shape

* update si

* more cleanup

* more small changes

* new shape finding logic

* hm

* check memory reads

* revert some changes

* optimistic commit

* some reverts

* fix rv32im costs

* update pr to work

* fix: shape panic handling

* fix: handle panic during shape vkey setup

* vk map

* fix

* impact

* push

* recursion shape

* do

* fix script

* generate back

* test

* print panic shapes

* fix recursion script

* make shape bigger

* shape update

* beq failure test

* cleaner types

* fix

* verify vk true

* changes

* addresses PR comment

* test shrink

* some comment fixes

* latest commit

* small shapes checkpoint

* new vk map

* vk map

* Revert "task(core): malicious trace generator support for failure test cases"

* update action

* update action

* update shapes

* new shapes

* perf: executor performance (#145)

* which events are exceeding

* coreairid

* cleanup -> 37 on mac

* more cleanup

* fix the test

* get script to compile
;

* reorder airs

* change default to auto

* comment nit

* Revert "perf: executor performance (#145)"

This reverts commit 7344448.

* fix: executor oom

* perf: executor performance (#145)

* which events are exceeding

* coreairid

* cleanup -> 37 on mac

* more cleanup

* fix the test

* get script to compile
;

* reorder airs

* change default to auto

* comment nit

* Revert "perf: executor performance (#145)"

This reverts commit af0b64e.

* Reapply "perf: executor performance (#145)"

This reverts commit ae08568.

* fix executor commit

* fix

* disable `sp1-recursion-compiler/debug` feature by default (#154)

* feat(recursion): print backtraces on failed assertions (#155)

* replace `TracedVec` with `Vec`

* add `DebugBacktrace` to dsl and executor instructions

* print backtraces with feature `"debug"` on division by zero

* add `--no-fail-fast` to CI. should revert when CI passes

* update gitignore

* clippy fix

* fix clippy

* update release

* update vk

* generate proofs in tests

* nit

* more cleanup

* Update crates/core/executor/src/profiler.rs

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* fix issues

* fix: zkvm workspace version (#166)

* fix ssz withdrawals (#165)

* refactor: sdk v2 (#1875)

Co-authored-by: Tamir Hemo <[email protected]>
Co-authored-by: Tej Qu Nair <[email protected]>
Co-authored-by: Kevin Jue <[email protected]>
Co-authored-by: John Guibas <[email protected]>
Co-authored-by: erabinov <[email protected]>
Co-authored-by: Eugene Rabinovich <[email protected]>
Co-authored-by: Yuwen Zhang <[email protected]>
Co-authored-by: N <[email protected]>

* separate out veriifer crate test

* fix format

* fix examples

* fix executor clippy

* changes

* modify

* fix sdk

* cleanup files a bit

* update groth16_vk

* fix shape better algo

* shape fix

* fix shape matching logic

* fix fix_shape

* new vk map

* new sdk in verifier crate

* feat(sdk): `get_proof_status`, `request`, `cycle_limit` (#1883)

* feat(sdk): latest network proto + error handling + fix examples (#1884)

* feat: use `B256` + `request_async` + bump `alloy` + fix CI (#1886)

* docs: add conventional commit guidelines to CONTRIBUTING.md (#1854)

* fix: remove stdin from SP1ProofWithPublicValues (#1714)

* fix: default tag

* feat: update book, imports

* fix

* use SP1_CIRCUIT_VERSION

* docs

* fix

* fix

* feat: change loc, fix later

* add

* add

* add

* fix

* update vk

* fix examples

* update vk

* lock

* fmt

---------

Co-authored-by: Kevin Jue <[email protected]>
Co-authored-by: Tamir Hemo <[email protected]>
Co-authored-by: Tej Qu Nair <[email protected]>
Co-authored-by: tamirhemo <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Chris Tian <[email protected]>
Co-authored-by: Bhargav Annem <[email protected]>
Co-authored-by: Yuwen Zhang <[email protected]>
Co-authored-by: rkm0959 <[email protected]>
Co-authored-by: erabinov <[email protected]>
Co-authored-by: Eugene Rabinovich <[email protected]>
Co-authored-by: N <[email protected]>
Co-authored-by: Ratan Kaliani <[email protected]>
Co-authored-by: Matt Stam <[email protected]>
Co-authored-by: Jev Björsell <[email protected]>
  • Loading branch information
16 people authored Dec 21, 2024
1 parent 4fe8144 commit 15b9b0f
Show file tree
Hide file tree
Showing 62 changed files with 1,432 additions and 586 deletions.
47 changes: 45 additions & 2 deletions .github/workflows/pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ jobs:
with:
command: test
toolchain: 1.81.0
args: --release --features native-gnark --no-fail-fast # TODO remove when CI checks always pass
args: --release --features native-gnark --workspace --exclude sp1-verifier
env:
RUSTFLAGS: -Copt-level=3 -Cdebug-assertions -Coverflow-checks=y -Cdebuginfo=0 -C target-cpu=native
RUST_BACKTRACE: 1
Expand Down Expand Up @@ -100,12 +100,55 @@ jobs:
with:
command: test
toolchain: 1.81.0
args: --release --features native-gnark --no-fail-fast # TODO remove when CI checks always pass
args: --release --features native-gnark --workspace --exclude sp1-verifier
env:
RUSTFLAGS: -Copt-level=3 -Cdebug-assertions -Coverflow-checks=y -Cdebuginfo=0 -C target-cpu=native
RUST_BACKTRACE: 1
SP1_DEV: 1

test-verifier:
name: Test Verifier Crate
runs-on:
[
runs-on,
runner=64cpu-linux-arm64,
spot=false,
disk=large,
"run-id=${{ github.run_id }}",
]
env:
CARGO_NET_GIT_FETCH_WITH_CLI: "true"
steps:
- name: Checkout sources
uses: actions/checkout@v4

- name: Setup CI
uses: ./.github/actions/setup

- name: Install SP1 toolchain from repo
run: |
cargo run -p sp1-cli -- prove install-toolchain
cd crates/cli
cargo install --locked --force --path .
cargo clean
- name: Run cargo check
uses: actions-rs/cargo@v1
with:
command: check
toolchain: 1.81.0
args: --all-targets --all-features

- name: Run cargo test
uses: actions-rs/cargo@v1
with:
command: test
toolchain: 1.81.0
args: --release --package sp1-verifier
env:
RUSTFLAGS: -Copt-level=3 -Cdebug-assertions -Coverflow-checks=y -Cdebuginfo=0 -C target-cpu=native
RUST_BACKTRACE: 1

lint:
name: Formatting & Clippy
runs-on: [runs-on, runner=16cpu-linux-x64, disk=large, "run-id=${{ github.run_id }}"]
Expand Down
21 changes: 20 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,25 @@ changes that are split across multiple commits.
That said, if you have a number of commits that are "checkpoints" and don't represent a single logical change, please
squash those together.

**Conventional Commit Messages and PR Titles:**
To ensure consistency and aid in automated tooling (such as changelog generation), please follow the
[Conventional Commits](https://www.conventionalcommits.org/) style for commit messages and PR titles. Prefix your commit
messages and PR titles with one of the following types:

- **feat:** for new features
- **fix:** for bug fixes
- **docs:** for documentation-only changes
- **refactor:** for code changes that neither fix a bug nor add a feature
- **chore:** for changes to the build process or auxiliary tools
- **test:** for adding or updating tests

For example:
- `docs: Update README with corrected links`
- `fix: Resolve race condition in event loop`

If your pull request title does not contain a valid prefix, automated checks may fail. You can amend your commit and
force-push to correct this before merging.

#### Opening the pull request

From within GitHub, opening a new pull request will present you with a template that should be filled out. Please try
Expand Down Expand Up @@ -162,4 +181,4 @@ _Adapted from the [Reth contributing guide](https://raw.githubusercontent.com/pa

[mcve]: https://stackoverflow.com/help/mcve

[hiding-a-comment]: https://help.github.com/articles/managing-disruptive-comments/#hiding-a-comment
[hiding-a-comment]: https://help.github.com/articles/managing-disruptive-comments/#hiding-a-comment
Loading

0 comments on commit 15b9b0f

Please sign in to comment.