Why memory usage is so high?

[doutv]

fork11+ support maximum total steps = 2^25

I run it on a 1TB memory cloud server and it eats more than 800GB+ memory!

This blog post benchmark polynomial commitment scheme Commit
winter can commit 2^31 elements on hpc6a(370GB memory). There is a large gap between 2^31 and 2^25.

Can someone explain why STARK proving in zkevm-prover consume such huge memory?

[doutv]

@rickb80 @fractasy
Could you answer this question?

[obynonwane]

The reason why zkevm prover consumes alot of memory can be attributed to a number of factors for starters it must emulate the entire Ethereum EVM, e.g Storage & Calldata, smart contract execution, Cryptographic primitives like Keccak256, so there are far more contraint on the zkevm prover as compared to some other circuits like the one in the blog post.

The zkevm Prover uses STARK which requires large FFT computation for polynomials evaluation, there is also alot of redundancy as a result, STARKs often consume significantly more memory than only SNARK-based systems for equivalent circuit sizes.

I also think the zkEVM prover is designed to handle a more general-purpose EVM emulator with a broader, less predictable set of operations. This some how reduces opportunities for optimizations like circuit specialization or memory reuse. Library like halo2 and plonky2 are highly optimised.

Another area to consider is that zkEVM implements proof recursion to scale its proofs. The downside is that recursion adds a blowup factor in memory usage, because each recursive step introduces additional memory requirements to process and verify earlier proofs. zkevm splitting workloads across threads introduces overhead of increased memory usage.

There is a permanent contraint for zkevm as, e.g like encoding of EVM specific rules, task like verifying cryptographic primitives (e.g Merkle trees, Keccak hashing), they exponentially increase memory usage.

cc @eduadiez @RogerTaule @fractasy