perf(l2): add prover benchmarks from L1 RPC downloaded blocks #1705

xqft · 2025-01-13T20:46:31Z

Motivation

We want to benchmark the L2 prover against rsp, using Ethereum mainnet/testnet blocks. For this we need to fetch state from an Ethereum chain.

Description

adds l2/prover/bench crate
adds state fetching functions from the JSON-RPC API (account, block, storage, proofs)
adds a RpcDB type to download and cache needed state during block execution
adds an execute() function to the Prover type
fixes state verification (via state and storage MPTs), verify_db()
fixes withdrawals not being processed when using ExecutionDB
fixes beacon root call state changes not being comitted when using ExecutionDB

github-actions · 2025-01-13T20:49:21Z

| File                                                                       | Lines | Diff |
+----------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/cmd/ethrex/cli.rs                          | 143   | -8   |
+----------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/cmd/ethrex/ethrex.rs                       | 404   | -24  |
+----------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/common/types/block.rs               | 648   | -1   |
+----------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/l2/proposer/prover_server.rs        | 459   | +8   |
+----------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/l2/prover/bench/src/cache.rs        | 23    | +23  |
+----------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/l2/prover/bench/src/constants.rs    | 35    | +35  |
+----------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/l2/prover/bench/src/lib.rs          | 3     | +3   |
+----------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/l2/prover/bench/src/main.rs         | 101   | +101 |
+----------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/l2/prover/bench/src/rpc/db.rs       | 416   | +416 |
+----------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/l2/prover/bench/src/rpc/mod.rs      | 319   | +319 |
+----------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/l2/prover/src/prover.rs             | 140   | +22  |
+----------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/l2/prover/zkvm/interface/src/lib.rs | 179   | +15  |
+----------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/l2/utils/prover/proving_systems.rs  | 127   | +4   |
+----------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/l2/utils/test_data_io.rs            | 88    | +3   |
+----------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/vm/backends/revm.rs                 | 498   | +17  |
+----------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/vm/db.rs                            | 279   | +104 |
+----------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/vm/execution_db.rs                  | 171   | -58  |
+----------------------------------------------------------------------------+-------+------+

Total lines added: +1070
Total lines removed: 91
Total lines changed: 1161

github-actions · 2025-02-10T17:48:47Z

Summary: 11318/14408 (78.55%)

Prague: 2373/2373 (100.00%)
Cancun: 3579/3579 (100.00%)
Shanghai: 221/221 (100.00%)
Paris: 62/62 (100.00%)
London: 39/39 (100.00%)
Berlin: 2/35 (5.71%)
Istanbul: 34/35 (97.14%)
Constantinople: 1685/2439 (69.09%)
Byzantium: 1795/2330 (77.04%)
Homestead: 612/1324 (46.22%)
Frontier: 149/742 (20.08%)

github-actions · 2025-02-10T17:51:21Z

Benchmark Results Comparison

PR Results

Benchmark Results: Factorial

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_Factorial`	231.6 ± 1.1	230.5	233.6	1.00
`levm_Factorial`	902.0 ± 8.9	894.8	917.5	3.90 ± 0.04

Benchmark Results: Factorial - Recursive

Command	Mean [s]	Min [s]	Max [s]	Relative
`revm_FactorialRecursive`	1.454 ± 0.062	1.354	1.577	1.00
`levm_FactorialRecursive`	15.687 ± 0.028	15.647	15.723	10.79 ± 0.46

Benchmark Results: Fibonacci

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_Fibonacci`	208.6 ± 3.2	206.5	216.8	1.00
`levm_Fibonacci`	901.9 ± 7.6	892.3	919.4	4.32 ± 0.08

Benchmark Results: ManyHashes

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_ManyHashes`	8.8 ± 0.7	8.5	10.8	1.00
`levm_ManyHashes`	18.5 ± 0.2	18.1	18.7	2.09 ± 0.17

Benchmark Results: BubbleSort

Command	Mean [s]	Min [s]	Max [s]	Relative
`revm_BubbleSort`	3.206 ± 0.026	3.191	3.274	1.00
`levm_BubbleSort`	6.205 ± 0.042	6.166	6.273	1.94 ± 0.02

Benchmark Results: ERC20 - Transfer

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_ERC20Transfer`	248.3 ± 1.1	247.1	250.6	1.00
`levm_ERC20Transfer`	547.4 ± 6.8	539.8	564.4	2.20 ± 0.03

Benchmark Results: ERC20 - Mint

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_ERC20Mint`	140.5 ± 0.5	139.8	141.2	1.00
`levm_ERC20Mint`	356.3 ± 2.6	353.7	361.1	2.54 ± 0.02

Benchmark Results: ERC20 - Approval

Command	Mean [s]	Min [s]	Max [s]	Relative
`revm_ERC20Approval`	1.052 ± 0.036	1.024	1.150	1.00
`levm_ERC20Approval`	2.052 ± 0.012	2.039	2.078	1.95 ± 0.07

Main Results

Benchmark Results: Factorial

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_Factorial`	240.1 ± 21.7	229.8	301.4	1.00
`levm_Factorial`	917.1 ± 41.3	896.3	1033.4	3.82 ± 0.39

Benchmark Results: Factorial - Recursive

Command	Mean [s]	Min [s]	Max [s]	Relative
`revm_FactorialRecursive`	1.429 ± 0.101	1.324	1.569	1.00
`levm_FactorialRecursive`	15.682 ± 0.028	15.627	15.731	10.97 ± 0.78

Benchmark Results: Fibonacci

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_Fibonacci`	208.2 ± 2.0	206.2	212.4	1.00
`levm_Fibonacci`	931.1 ± 55.8	884.2	1045.5	4.47 ± 0.27

Benchmark Results: ManyHashes

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_ManyHashes`	8.7 ± 0.1	8.6	8.8	1.00
`levm_ManyHashes`	18.4 ± 0.5	18.0	19.7	2.12 ± 0.06

Benchmark Results: BubbleSort

Command	Mean [s]	Min [s]	Max [s]	Relative
`revm_BubbleSort`	3.223 ± 0.014	3.204	3.243	1.00
`levm_BubbleSort`	6.170 ± 0.045	6.117	6.249	1.91 ± 0.02

Benchmark Results: ERC20 - Transfer

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_ERC20Transfer`	249.2 ± 2.4	246.5	253.6	1.00
`levm_ERC20Transfer`	543.6 ± 2.6	540.6	549.0	2.18 ± 0.02

Benchmark Results: ERC20 - Mint

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_ERC20Mint`	140.7 ± 0.7	139.7	141.9	1.00
`levm_ERC20Mint`	356.8 ± 4.9	352.7	367.8	2.54 ± 0.04

Benchmark Results: ERC20 - Approval

Command	Mean [s]	Min [s]	Max [s]	Relative
`revm_ERC20Approval`	1.038 ± 0.005	1.029	1.044	1.00
`levm_ERC20Approval`	2.054 ± 0.059	2.022	2.221	1.98 ± 0.06

This reverts commit ec24436.

xqft added 2 commits January 14, 2025 12:31

add indexdb

94fb3e3

rename InnerIndexDB to AuxIndexDB

2847cf8

xqft mentioned this pull request Jan 14, 2025

fix(l2): replace ExecutionDB from_exec() impl to use a CacheDB based approach #1709

Merged

xqft added 4 commits January 15, 2025 11:06

add IndexDBError

89ba179

change Revm EVM error to ours

f5192c4

replace from_exec impl with indexdb

9616b7d

Merge branch 'main' into l2/touched_state

9d97192

xqft force-pushed the l2/rpc_bench branch from eb514ad to 98bab67 Compare January 15, 2025 20:46

xqft changed the base branch from main to l2/touched_state January 15, 2025 20:46

xqft added 17 commits January 17, 2025 11:39

WIP add PreExecDB

253ab7f

WIP add proofs in from_store()

966706e

WIP remove internal mutability

2135a08

refactor

eabd6a3

replace PreExecDB with CacheDB

9ffbc44

nits

a308649

fix filter

b0d214f

add nonce and balance validations during pre-exec

1b64809

put proofs inside executiondb

dd75353

fix errors

389ddab

remove optiona_balance_check feature

fec82ef

remove unused errors

0405835

fix filter

987e359

add db verification

4a0a715

remove unused imports

eb98e53

update sp1 prover

892060a

Merge branch 'main' into l2/touched_state

d9aee43

xqft force-pushed the l2/rpc_bench branch from 98bab67 to c756be3 Compare January 20, 2025 20:56

xqft mentioned this pull request Jan 21, 2025

perf(l2): rpc comparison bench #1360

Closed

remove revm types

6f2169c

xqft added 6 commits February 10, 2025 12:54

add env vars

fae29b3

downgrade sp1

0718fe1

add prove rules

b5c239e

disable default features for core and vm

d639371

remove c-kzg dep from zkvm programs

79b3b7a

fix imports

95fac58

xqft added 15 commits February 10, 2025 14:57

fix elapsed time disp

e9173dc

bump risc0

e560183

add support for risc0

febe737

add block num env var

075ac72

remove exclusive compile err

77f88dc

fix prev commit

bc41c07

fix cache deser

ec24436

Revert "fix cache deser"

7574ddd

This reverts commit ec24436.

comment out requests_hash conditional serde

0838863

bump sp1 version

0a4314f

serialize into json

a86e66b

change file extension

58a741b

create file

4c10ab7

better final print

71913fa

Merge branch 'main' into l2/rpc_bench

8cc38b9

xqft marked this pull request as ready for review February 11, 2025 17:01

xqft requested a review from a team as a code owner February 11, 2025 17:01

Oppen and others added 5 commits February 12, 2025 14:07

Merge remote-tracking branch 'origin/main' into l2/rpc_bench

603306f

fix build (fails perf-sp1 and perf-risc0 tests)

005f619

fix test build

9ae47e1

fix beacon root call executing on perf_zkvm tests

e576b2b

fix prover mode name

bf0779c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf(l2): add prover benchmarks from L1 RPC downloaded blocks #1705

perf(l2): add prover benchmarks from L1 RPC downloaded blocks #1705

xqft commented Jan 13, 2025 •

edited

Loading

github-actions bot commented Jan 13, 2025 •

edited

Loading

github-actions bot commented Feb 10, 2025 •

edited

Loading

github-actions bot commented Feb 10, 2025 •

edited

Loading

perf(l2): add prover benchmarks from L1 RPC downloaded blocks #1705

Are you sure you want to change the base?

perf(l2): add prover benchmarks from L1 RPC downloaded blocks #1705

Conversation

xqft commented Jan 13, 2025 • edited Loading

github-actions bot commented Jan 13, 2025 • edited Loading

github-actions bot commented Feb 10, 2025 • edited Loading

github-actions bot commented Feb 10, 2025 • edited Loading

Benchmark Results Comparison

PR Results

Benchmark Results: Factorial

Benchmark Results: Factorial - Recursive

Benchmark Results: Fibonacci

Benchmark Results: ManyHashes

Benchmark Results: BubbleSort

Benchmark Results: ERC20 - Transfer

Benchmark Results: ERC20 - Mint

Benchmark Results: ERC20 - Approval

Main Results

Benchmark Results: Factorial

Benchmark Results: Factorial - Recursive

Benchmark Results: Fibonacci

Benchmark Results: ManyHashes

Benchmark Results: BubbleSort

Benchmark Results: ERC20 - Transfer

Benchmark Results: ERC20 - Mint

Benchmark Results: ERC20 - Approval

xqft commented Jan 13, 2025 •

edited

Loading

github-actions bot commented Jan 13, 2025 •

edited

Loading

github-actions bot commented Feb 10, 2025 •

edited

Loading

github-actions bot commented Feb 10, 2025 •

edited

Loading