[saya] Scheduler (starkware-libs#1917)

* proof scheduler * cleared leftovers * feature fix * Deserialization and tests fix * uncommented apply diffs logic * genesis block fix * fetching last block * moved fetching to separate function * process_block no async * Adding traces and infos for async functions * made dojo core work with merger program * minor logs adjustments in saya * final world contract * preparation for http prover * proving example data by * serializing arguments * debug print * only one program in verifier * preparing args for prover * batch size arg * moved extraction of data from proof to separate module * minor cleanup * updated bin/scheduler * laziness fix and update prover sdk * half done * cairo 1 behind flag * added cairo 0 differ and merger * support for 2 programs in `upgrade_state` * end 2 end saya readme * data extraction from proof * saya working with cairo 0 * updated prover and readme * added a clause about keygen * updated port in readme * clippy and formatting * ignored heavy tests * moved readme to bin * updated stone prover image * made tests pass * removed unneeded files and fixed compilation * formatting after rebase * better fetch parallelization * fixed valid ignoring of not l1_handler * fixed most tests and made compiled programs one line * cairo import formatting * using recursive layout * changed docker images to latest * Update examples/spawn-and-move/Scarb.toml Co-authored-by: glihm <[email protected]> * most of pr comments * keeping track of nonce to avoid sleep * rest of PR comments * made changes to fix the tests * formatting * fixed all the tests * final formatting * rebuilt artifacts * fix: typo in README * fixed duplicated proving state update * fix: remove cargo warning for dual license source * fix: reword some commands + fix options for saya * fix: update README * fix: ensure model get fails if no key is provided * fix: use --bin instead of -p --------- Co-authored-by: Mateusz Zając <[email protected]> Co-authored-by: Mateusz <[email protected]> Co-authored-by: Mateusz Zając <[email protected]> Co-authored-by: glihm <[email protected]>
reilabs · Jun 11, 2024 · d3c3351 · d3c3351
1 parent 9205009
commit d3c3351
Show file tree

Hide file tree

Showing 46 changed files with 2,429 additions and 367 deletions.
diff --git a/Cargo.lock b/Cargo.lock
diff --git a/Cargo.toml b/Cargo.toml
@@ -5,6 +5,7 @@ members = [
     "bin/dojo-language-server",
     "bin/katana",
     "bin/saya",
+    "bin/scheduler",
     "bin/sozo",
     "bin/torii",
     "crates/benches",
@@ -189,7 +190,6 @@ tower-http = "0.4.4"
 tracing = "0.1.34"
 tracing-subscriber = { version = "0.3.16", features = [ "env-filter", "json" ] }
 url = { version = "2.4.0", features = [ "serde" ] }
-
 rstest = "0.18.2"
 rstest_reuse = "0.6.0"
 

diff --git a/bin/saya/Cargo.toml b/bin/saya/Cargo.toml
@@ -20,3 +20,7 @@ tokio.workspace = true
 tracing-subscriber.workspace = true
 tracing.workspace = true
 url.workspace = true
+
+[dev-dependencies]
+cairo-proof-parser.workspace = true
+starknet-crypto.workspace = true
diff --git a/bin/saya/README.md b/bin/saya/README.md
@@ -1,18 +1,21 @@
-
 # Saya Executable Documentation
 
 This documentation outlines the operation of the Saya executable, a CLI-based service designed to interact with blockchain components for state management and updates. Saya supports operations on Celestia nodes and integrates with Katana blocks to provide a streamlined blockchain interaction process.
 
 ## Key Features
+
 - **Celestia Node Integration**: Allows publishing state updates to a Celestia node (WIP).
 - **Katana Block Fetching**: Saya can fetch blocks from Katana, aiding in local blockchain simulations and testing.
 
 ## Prerequisites
+
 Ensure you have the following set up:
+
 - Rust and Cargo installed on your system.
 - Access to Celestia and/or Katana node URLs if needed.
 
 ## Basic Usage Example
+
 Below is a command-line example that demonstrates how to run the Saya executable with necessary parameters:
 
 ```bash
@@ -21,38 +24,160 @@ cargo run --bin saya -- --rpc-url http://localhost:5050 --da-chain celestia --ce
 
 ## Detailed Workflow
 
-### 1. Deploy a New World Contract
-First, deploy a new smart contract using the Saya framework:
-```bash
-cargo run -r -p sozo -- build --manifest-path examples/spawn-and-move/Scarb.toml
+1. Prepare fact registry contract
+
+   Declare or use already declared `class-hash`: `0x7f6076572e04d7182a1c5c9f1f4c15aafcb069b1bfdb3de4d7c9e47c99deeb4`.
+
+   Deploy or use already deployed `contract`: `0x217746a5f74c2e5b6fa92c97e902d8cd78b1fabf1e8081c4aa0d2fe159bc0eb`.
+
+   In the repository https://github.com/HerodotusDev/integrity run
 
-cargo run -r -p sozo -- migrate apply --manifest-path examples/spawn-and-move/Scarb.toml --rpc-url <> --fee-estimate-multiplier 1000 --private-key <> --account-address <> --name saya-world-v1
+```bash
+    fact_registry/1-declare.sh # extract `class-hash`
+    fact_registry/1-deploy.sh <CLASS_HASH> # use at <FACT_REGISTRY>
 ```
 
-### 2. Initialize a Local Katana Instance
-Start a local instance of Katana configured to work with the newly deployed contract:
+2. Spawn world
+
+You must choose a world's name as you may deploy the exact same code as an other person trying this example. The world's name must fit into 31 characters.
+
+**IMPORTANT NOTE:**
+At the moment until a bug is fixed, you must comment out the `world_address` that is present into the `examples/spawn-and-move/Scarb.toml` file to allow the use of the `--name` flag.
+
 ```bash
-cargo run -r -p katana -- -b 30000 --rpc-url <> --fork-block-number <block number after world deployment>
+cargo run -r --bin sozo -- \
+    build \
+    --manifest-path examples/spawn-and-move/Scarb.toml
+
+cargo run -r --bin sozo -- \
+    migrate apply \
+    --manifest-path examples/spawn-and-move/Scarb.toml \
+    --rpc-url <SEPOLIA_ENDPOINT> \
+    --private-key <SEPOLIA_PRIVATE_KEY> \
+    --account-address <SEPOLIA_ACCOUNT_ADDRESS> \
+    --fee-estimate-multiplier 20 \
+    --name <WORLD_NAME>
 ```
+Once the migration is done, please take note of the address of the world as it will be re-used in the commands below.
+
+1. Set world configs
 
-### 3. Launch the Saya Service
-Execute the Saya process to interact with the blockchain network:
 ```bash
-cargo run -r --bin saya -- --rpc-url http://localhost:5050 --registry 0x217746a5f74c2e5b6fa92c97e902d8cd78b1fabf1e8081c4aa0d2fe159bc0eb --world <> --start-block <>
+cargo run -r --bin sozo -- \
+    execute <WORLD_ADDRESS> set_differ_program_hash \
+    -c 0xa73dd9546f9858577f9fdbe43fd629b6f12dc638652e11b6e29155f4c6328 \
+    --manifest-path examples/spawn-and-move/Scarb.toml \
+    --rpc-url <SEPOLIA_ENDPOINT> \
+    --private-key <SEPOLIA_PRIVATE_KEY> \
+    --account-address <SEPOLIA_ACCOUNT_ADDRESS> \
+    --fee-estimate-multiplier 20 \
+    --world <WORLD_ADDRESS> \
+    --wait
+
+cargo run -r --bin sozo -- \
+    execute <WORLD_ADDRESS> set_merger_program_hash \
+    -c 0xc105cf2c69201005df3dad0050f5289c53d567d96df890f2142ad43a540334 \
+    --manifest-path examples/spawn-and-move/Scarb.toml \
+    --rpc-url <SEPOLIA_ENDPOINT> \
+    --private-key <SEPOLIA_PRIVATE_KEY> \
+    --account-address <SEPOLIA_ACCOUNT_ADDRESS> \
+    --fee-estimate-multiplier 20 \
+    --world <WORLD_ADDRESS> \
+    --wait
+
+cargo run -r --bin sozo -- \
+    execute <WORLD_ADDRESS> set_facts_registry \
+    -c 0x217746a5f74c2e5b6fa92c97e902d8cd78b1fabf1e8081c4aa0d2fe159bc0eb \
+    --manifest-path examples/spawn-and-move/Scarb.toml \
+    --rpc-url <SEPOLIA_ENDPOINT> \
+    --private-key <SEPOLIA_PRIVATE_KEY> \
+    --account-address <SEPOLIA_ACCOUNT_ADDRESS> \
+    --fee-estimate-multiplier 20 \
+    --world <WORLD_ADDRESS> \
+    --wait
 ```
 
-Currently the Stone prover in use is optimized for AMD64. Hence, running it on a ARM64 machine will be relatively slow or even not compatible if emulation is now available.
-If you can't run Saya on a AMD64 machine, you may choose to use the HTTP wrapper currently proposed by Visoft.
+4. Start katana
+
+Start a local instance of Katana configured to work with the newly deployed contract. You should wait your world to be integrated into the latest block (and not the pending).
+Once block in which the transaction that deploys the world is mined, you can start `katana` in forking mode.
+
 ```bash
-cargo run -r --bin saya -- --rpc-url http://localhost:5050 --prover-url http://prover.visoft.dev:3618/prove/state-diff-commitment --registry 0x217746a5f74c2e5b6fa92c97e902d8cd78b1fabf1e8081c4aa0d2fe159bc0eb --world <> --start-block <>
+cargo run -r --bin katana -- \
+    --rpc-url <SEPOLIA_ENDPOINT> \
+    --fork-block-number <LATEST_BLOCK>
 ```
 
-### 4. Modify the World State Using `sozo`
+5. Run transactions on `katana`
+
 Finally, modify the state of the world using specific actions:
+
+```bash
+cargo run -r --bin sozo -- execute dojo_examples::actions::actions spawn \
+    --manifest-path examples/spawn-and-move/Scarb.toml \
+    --rpc-url http://localhost:5050 \
+    --private-key <SEPOLIA_PRIVATE_KEY> \
+    --account-address <SEPOLIA_ACCOUNT_ADDRESS> \
+    --world <WORLD_ADDRESS> \
+    --wait
+```
+
+Before running `saya`, we can check the actual value for some models on Sepolia, to then see them updated by the proof being verified and the state of the world being updated.
+In the `spawn-and-move` example, the `Position` model is used to store some data associated with the player,
+being the contract address of the contract that called `spawn` (hence, your account address).
+By default on Sepolia, it should be set like to unknown position, being like:
+
+```bash
+cargo run -r --bin sozo -- model get Position <ACCOUNT_ADDRESS> \
+    --manifest-path examples/spawn-and-move/Scarb.toml \
+    --rpc-url <SEPOLIA_ENDPOINT> \
+    --world <WORLD_ADDRESS>
+```
+```json
+// Expected on Sepolia as we've executed the transaction on the Katana shard.
+{
+    player          : <SEPOLIA_ACCOUNT_ADDRESS>,
+    vec             : {
+        x               : 0,
+        y               : 0
+    }
+}
+
+// Expected on Katana.
+{
+    player          : <SEPOLIA_ACCOUNT_ADDRESS>,
+    vec             : {
+        x               : 10,
+        y               : 10
+    }
+}
+```
+
+6. Run saya
+
+The <PROVER_URL> could be `http://prover.visoft.dev:3618` if you have a registered key or a link to a self hosted instance of `https://github.com/neotheprogramist/http-prover`.
+The <PROVER_KEY> is the private key produced by `keygen` installed with `cargo install --git https://github.com/neotheprogramist/http-prover keygen`. Pass the public key to server operator or the prover program.
+
+If you are on an `amd64` architecture, go ahead and run the `http-prover` locally to see how it works and run this whole pipeline locally.
+If not (this includes Apple Silicon), some emulation will take place to run the prover on your machine, and this is very very slow.
+
+It's important that the `--start-block` of Saya is the first block produced by Katana as for now Katana is not fetching events from the forked network.
+
+**IMPORTANT NOTE:**
+For now, please add your account address and account private key in `saya/core/src/dojo_os/mod.rs` as those parameters are still not exposed currently. As you are using `cargo run`, it will rebuild with your account configuration before running `saya`.
+
 ```bash
-cargo run -r -p sozo -- execute --rpc-url http://localhost:5050 --private-key <> --account-address <> --world <> dojo_examples::actions::actions spawn
+cargo run -r --bin saya -- \
+    --rpc-url http://localhost:5050 \
+    --registry <FACT_REGISTRY> \
+    --world <WORLD_ADDRESS> \
+    --url <PROVER_URL> \
+    --private-key <PROVER_KEY> \
+    --start-block <FORKED_BLOCK_PLUS_1>
 ```
 
 After this command, Saya will pick up the blocks with transactions, generate the proof for the state transition, and send it to the base layer world contract.
 
+Once the world on Sepolia is updated, you can issue again the `model get` command as seen before, and you should see the `katana` shard state reflected on Sepolia.
+
 Ensure to replace placeholders (`<>`) with appropriate values for your configuration and environment. This documentation provides a comprehensive overview for developers and operators to effectively utilize the Saya service in blockchain applications.
diff --git a/bin/saya/programs/cairo0differ.json b/bin/saya/programs/cairo0differ.json
diff --git a/bin/saya/programs/cairo0merger.json b/bin/saya/programs/cairo0merger.json
diff --git a/bin/saya/src/args/mod.rs b/bin/saya/src/args/mod.rs
@@ -6,7 +6,7 @@ use std::path::PathBuf;
 use clap::Parser;
 use saya_core::data_availability::celestia::CelestiaConfig;
 use saya_core::data_availability::DataAvailabilityConfig;
-use saya_core::SayaConfig;
+use saya_core::{ProverAccessKey, SayaConfig};
 use tracing::Subscriber;
 use tracing_subscriber::{fmt, EnvFilter};
 use url::Url;
@@ -32,7 +32,19 @@ pub struct SayaArgs {
     #[arg(long)]
     #[arg(value_name = "PROVER URL")]
     #[arg(help = "The Prover URL for remote proving.")]
-    pub prover_url: Url,
+    pub url: Url,
+
+    /// Specify the Prover Key.
+    #[arg(long)]
+    #[arg(value_name = "PROVER KEY")]
+    #[arg(help = "An authorized prover key for remote proving.")]
+    pub private_key: String,
+
+    #[arg(long)]
+    #[arg(value_name = "STORE PROOFS")]
+    #[arg(help = "When enabled all proofs are saved as a file.")]
+    #[arg(default_value_t = false)]
+    pub store_proofs: bool,
 
     /// Enable JSON logging.
     #[arg(long)]
@@ -50,6 +62,10 @@ pub struct SayaArgs {
     #[arg(short, long, default_value = "0")]
     pub start_block: u64,
 
+    #[arg(short, long, default_value = "1")]
+    #[arg(help = "The number of blocks to be merged into a single proof.")]
+    pub batch_size: usize,
+
     #[command(flatten)]
     #[command(next_help_heading = "Data availability options")]
     pub data_availability: DataAvailabilityOptions,
@@ -117,10 +133,17 @@ impl TryFrom<SayaArgs> for SayaConfig {
                 None => None,
             };
 
+            let prover_key = ProverAccessKey::from_hex_string(&args.private_key).map_err(|e| {
+                Box::new(std::io::Error::new(std::io::ErrorKind::InvalidInput, e.to_string()))
+            })?;
+
             Ok(SayaConfig {
                 katana_rpc: args.rpc_url,
-                prover_url: args.prover_url,
+                url: args.url,
+                private_key: prover_key,
+                store_proofs: args.store_proofs,
                 start_block: args.start_block,
+                batch_size: args.batch_size,
                 data_availability: da_config,
                 world_address: args.proof.world_address,
                 fact_registry_address: args.proof.fact_registry_address,
@@ -145,9 +168,13 @@ mod tests {
         let args = SayaArgs {
             config_file: Some(config_file_path.clone()),
             rpc_url: Url::parse("http://localhost:5050").unwrap(),
-            prover_url: Url::parse("http://localhost:5050").unwrap(),
+            url: Url::parse("http://localhost:5050").unwrap(),
+            private_key: "0xd0fa91f4949e9a777ebec071ca3ca6acc1f5cd6c6827f123b798f94e73425027"
+                .into(),
+            store_proofs: true,
             json_log: false,
             start_block: 0,
+            batch_size: 4,
             data_availability: DataAvailabilityOptions {
                 da_chain: None,
                 celestia: CelestiaOptions {
@@ -165,7 +192,13 @@ mod tests {
         let config: SayaConfig = args.try_into().unwrap();
 
         assert_eq!(config.katana_rpc.as_str(), "http://localhost:5050/");
-        assert_eq!(config.prover_url.as_str(), "http://localhost:1234/");
+        assert_eq!(config.url.as_str(), "http://localhost:1234/");
+        assert_eq!(config.batch_size, 4);
+        assert_eq!(
+            config.private_key.signing_key_as_hex_string(),
+            "0xd0fa91f4949e9a777ebec071ca3ca6acc1f5cd6c6827f123b798f94e73425027"
+        );
+        assert!(!config.store_proofs);
         assert_eq!(config.start_block, 0);
         if let Some(DataAvailabilityConfig::Celestia(celestia_config)) = config.data_availability {
             assert_eq!(celestia_config.node_url.as_str(), "http://localhost:26657/");

diff --git a/bin/saya/src/args/test_saya_config_file.json b/bin/saya/src/args/test_saya_config_file.json
@@ -1,6 +1,9 @@
 {
   "katana_rpc": "http://localhost:5050",
-  "prover_url": "http://localhost:1234",
+  "url": "http://localhost:1234",
+  "private_key": "0xd0fa91f4949e9a777ebec071ca3ca6acc1f5cd6c6827f123b798f94e73425027",
+  "store_proofs": false,
+  "batch_size": 4,
   "world_address": "0x332b8ff41b1b026991fa9b7f0ec352909f8bc33416b65a80527edc988a9b082",
   "fact_registry_address": "0x217746a5f74c2e5b6fa92c97e902d8cd78b1fabf1e8081c4aa0d2fe159bc0eb",
   "start_block": 0,

diff --git a/bin/saya/src/main.rs b/bin/saya/src/main.rs
@@ -6,6 +6,9 @@ use tokio::signal::ctrl_c;
 
 mod args;
 
+#[cfg(test)]
+mod tests;
+
 use args::SayaArgs;
 
 #[tokio::main]
@@ -54,7 +57,7 @@ CONFIGURATION
     if let Some(da_config) = &config.data_availability {
         println!(
             r"
-DATA AVAILBILITY
+DATA AVAILABILITY
 ==================
 {da_config}
     ",