Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(tests): Gas oracle tests #1567

Merged
merged 67 commits into from
Apr 9, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
a15cd2c
[WIP] Gas oracle to convert native token price
jrchatruc Feb 1, 2024
dcd2721
zk fmt
jrchatruc Feb 1, 2024
c8626a7
Connect with fetcher with config data to get the host and poll interval
jrchatruc Feb 1, 2024
06ec84d
Remove unused fields
jrchatruc Feb 1, 2024
5c6b647
add erc20 fetcher to gas adjuster
juan518munoz Feb 8, 2024
9a055f2
fix env vars names, remove unused variable
Feb 16, 2024
a16252b
remove cast to u64
Feb 16, 2024
4fe5b7a
remove unused field from ZkSyncStateKeeper
Feb 16, 2024
b7cbd70
Merge remote-tracking branch 'upstream/kl-factory' into gas-oracle
jrchatruc Feb 20, 2024
c3a8d80
Merge remote-tracking branch 'upstream/kl-factory' into gas-oracle
jrchatruc Feb 20, 2024
5de1233
Use = &mut instead of ref mut
jrchatruc Feb 26, 2024
773ff7a
Fetch the conversion rate on Fetcher initialization to avoid incorrec…
jrchatruc Feb 26, 2024
3cb3e5c
Merge remote-tracking branch 'upstream/kl-factory' into gas-oracle
jrchatruc Feb 26, 2024
fd3e7d2
Add http client as a field to the native erc20 fetcher
jrchatruc Feb 27, 2024
6838588
Rename NativeErc20 stuff to be NativeToken instead
jrchatruc Feb 27, 2024
2ebe86b
Merge remote-tracking branch 'upstream/lambdaclass_gas_oracle' into c…
jrchatruc Feb 27, 2024
67286bc
initial commit
juan518munoz Feb 28, 2024
08e8b5a
remove comment
juan518munoz Feb 28, 2024
7ca95b8
remove needless variable
juan518munoz Feb 28, 2024
0315e46
initial commit
juan518munoz Feb 28, 2024
c79dccb
initial commit
juan518munoz Feb 28, 2024
b6367ad
rmv unwrap
juan518munoz Feb 28, 2024
252312a
improve err handling
juan518munoz Feb 28, 2024
83dd481
Merge pull request #1263 from lambdaclass/cache-http-client
jrchatruc Feb 29, 2024
e91c27c
Merge pull request #1272 from lambdaclass/gas-oracle-noop-conversion-…
jrchatruc Feb 29, 2024
7214d5e
Merge branch 'lambdaclass_gas_oracle' into gas-oracle-change-err-type
juan518munoz Feb 29, 2024
7b33934
store native token address in fetcher config, use it in the conversio…
Feb 29, 2024
e4fb1c4
add mission EOF
Feb 29, 2024
6d114f3
remove print
Feb 29, 2024
3d4addd
Merge pull request #1281 from lambdaclass/gas-oracle-change-err-type
jrchatruc Feb 29, 2024
97f9eec
Merge branch 'lambdaclass_gas_oracle' into gas-oracle-imprv-fetcher-err
juan518munoz Feb 29, 2024
5eb92bc
impl ErrorReporter
juan518munoz Feb 29, 2024
13b84d0
use Address struct for token address
Mar 1, 2024
7f61a38
move struct & add context to err
juan518munoz Mar 1, 2024
a98a0b7
add context to init
juan518munoz Mar 1, 2024
e182381
add conversion rate component
Mar 1, 2024
6bb64ef
apply zk fmt
Mar 1, 2024
e7241fc
add token address to config
Mar 1, 2024
bfae1c6
Add missing trait
Mar 1, 2024
7077b40
add token address to config
Mar 1, 2024
3ab731b
Merge pull request #1301 from lambdaclass/add_token_address_to_api_su…
jrchatruc Mar 4, 2024
1154e54
merge lambdaclass_gas_oracle
juan518munoz Mar 4, 2024
79a8eb2
Merge pull request #1282 from lambdaclass/gas-oracle-imprv-fetcher-err
jrchatruc Mar 4, 2024
984e052
Merge branch 'lambdaclass_gas_oracle' into conversion_rate_component
juan518munoz Mar 4, 2024
dd11e4a
fix up code
juan518munoz Mar 4, 2024
1a20f2a
[WIP] Remove native token fetcher singleton
jrchatruc Mar 5, 2024
3103dca
Fix compilation (add missing await)
jrchatruc Mar 5, 2024
0e2f572
Merge remote-tracking branch 'upstream/lambdaclass_gas_oracle' into r…
jrchatruc Mar 5, 2024
cf5f2ff
fix stop handle for converstion rate api
juan518munoz Mar 6, 2024
46b6ef4
add logging for conversion rate
juan518munoz Mar 6, 2024
6e0c812
Remove unnecessary alert_spawned boolean
jrchatruc Mar 7, 2024
13cf343
Revert "Remove unnecessary alert_spawned boolean"
jrchatruc Mar 8, 2024
fbf66b1
Use a mutex for the errorreporter
jrchatruc Mar 8, 2024
a19be60
Revert changes to .init.env
jrchatruc Mar 8, 2024
d99e9bf
rmv flask app, rename component & refactor func
juan518munoz Mar 11, 2024
8d37924
rename module
juan518munoz Mar 11, 2024
feaf132
Merge pull request #1319 from lambdaclass/conversion_rate_component
jrchatruc Mar 11, 2024
3fff7d9
Merge remote-tracking branch 'upstream/lambdaclass_gas_oracle' into r…
jrchatruc Mar 11, 2024
ba7762c
Merge pull request #1360 from lambdaclass/remove-token-fetcher-singleton
jrchatruc Mar 11, 2024
d81aef0
Merge branch 'kl-factory' into lambdaclass_gas_oracle
jrchatruc Mar 12, 2024
5680df8
zk fmt
juan518munoz Mar 19, 2024
47c444f
Merge remote-tracking branch 'matter-labs/kl-factory' into gas-oracle
juan518munoz Mar 20, 2024
b520744
rename native token to base token
juan518munoz Mar 25, 2024
29289c9
fix env load
juan518munoz Mar 25, 2024
7c8ec45
add gas oracle documentation
Mar 27, 2024
2a7761d
fmt
Mar 27, 2024
55ae89f
Merge branch 'kl-factory' into gas-oracle
fkrause98 Mar 27, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions core/bin/zksync_server/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ use clap::Parser;
use zksync_config::{
configs::{
api::{HealthCheckConfig, MerkleTreeApiConfig, Web3JsonRpcConfig},
base_token_fetcher::BaseTokenFetcherConfig,
chain::{
CircuitBreakerConfig, MempoolConfig, NetworkConfig, OperationsManagerConfig,
StateKeeperConfig,
Expand Down Expand Up @@ -139,6 +140,7 @@ async fn main() -> anyhow::Result<()> {
eth_watch_config: ETHWatchConfig::from_env().ok(),
gas_adjuster_config: GasAdjusterConfig::from_env().ok(),
object_store_config: ObjectStoreConfig::from_env().ok(),
base_token_fetcher_config: BaseTokenFetcherConfig::from_env().ok(),
consensus_config: config::read_consensus_config().context("read_consensus_config()")?,
},
};
Expand Down
9 changes: 9 additions & 0 deletions core/lib/config/src/configs/base_token_fetcher.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
use serde::Deserialize;
use zksync_basic_types::Address;

#[derive(Debug, Clone, Deserialize, PartialEq)]
pub struct BaseTokenFetcherConfig {
pub poll_interval: u64,
pub host: String,
pub token_address: Address,
}
2 changes: 2 additions & 0 deletions core/lib/config/src/configs/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
pub use self::{
alerts::AlertsConfig,
api::ApiConfig,
base_token_fetcher::BaseTokenFetcherConfig,
contract_verifier::ContractVerifierConfig,
contracts::ContractsConfig,
database::{DBConfig, PostgresConfig},
Expand All @@ -23,6 +24,7 @@ pub use self::{

pub mod alerts;
pub mod api;
pub mod base_token_fetcher;
pub mod chain;
pub mod contract_verifier;
pub mod contracts;
Expand Down
4 changes: 2 additions & 2 deletions core/lib/dal/src/blocks_dal.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2214,7 +2214,7 @@ impl BlocksDal<'_, '_> {
)
WHERE
l1_batch_number IS NULL
AND fee_account_address = '\x0000000000000000000000000000000000000000'::bytea
AND fee_account_address = 'x0000000000000000000000000000000000000000'::bytea
"#
)
.execute(self.storage.conn())
Expand Down Expand Up @@ -2254,7 +2254,7 @@ impl BlocksDal<'_, '_> {
WHERE
l1_batches.number = miniblocks.l1_batch_number
AND miniblocks.number BETWEEN $1 AND $2
AND miniblocks.fee_account_address = '\x0000000000000000000000000000000000000000'::bytea
AND miniblocks.fee_account_address = 'x0000000000000000000000000000000000000000'::bytea
"#,
i64::from(numbers.start().0),
i64::from(numbers.end().0)
Expand Down
9 changes: 9 additions & 0 deletions core/lib/env_config/src/base_token_fetcher.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
use zksync_config::configs::base_token_fetcher::BaseTokenFetcherConfig;

use crate::{envy_load, FromEnv};

impl FromEnv for BaseTokenFetcherConfig {
fn from_env() -> anyhow::Result<Self> {
envy_load("base_token_fetcher", "BASE_TOKEN_FETCHER_")
}
}
1 change: 1 addition & 0 deletions core/lib/env_config/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ use serde::de::DeserializeOwned;

mod alerts;
mod api;
pub mod base_token_fetcher;
mod chain;
mod contract_verifier;
mod contracts;
Expand Down
26 changes: 26 additions & 0 deletions core/lib/protobuf_config/src/base_token_fetcher.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
use anyhow::{Context as _, Ok};
use zksync_config::configs;
use zksync_protobuf::{repr::ProtoRepr, required};

use crate::{parse_h160, proto::base_token_fetcher as proto};

impl ProtoRepr for proto::BaseTokenFetcher {
type Type = configs::BaseTokenFetcherConfig;
fn read(&self) -> anyhow::Result<Self::Type> {
Ok(Self::Type {
poll_interval: *required(&self.poll_interval).context("poll_interval")?,
host: required(&self.host).context("host")?.clone(),
token_address: required(&self.token_address)
.and_then(|a| parse_h160(a))
.context("fee_account_addr")?,
})
}

fn build(this: &Self::Type) -> Self {
Self {
poll_interval: Some(this.poll_interval),
host: Some(this.host.clone()),
token_address: Some(this.token_address.as_bytes().into()),
}
}
}
1 change: 1 addition & 0 deletions core/lib/protobuf_config/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

mod alerts;
mod api;
mod base_token_fetcher;
mod chain;
mod contract_verifier;
mod contracts;
Expand Down
9 changes: 9 additions & 0 deletions core/lib/protobuf_config/src/proto/base_token_fetcher.proto
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
syntax = "proto3";

package zksync.config.base_token_fetcher;

message BaseTokenFetcher {
optional uint64 poll_interval = 1;
optional string host = 2;
optional bytes token_address = 3;
}
145 changes: 145 additions & 0 deletions core/lib/zksync_core/src/base_token_fetcher/mod.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
use std::{cmp::min, sync::Arc};

use anyhow::Context;
use async_trait::async_trait;
use hex::ToHex;
use metrics::atomics::AtomicU64;
use tokio::sync::Mutex;
use zksync_config::configs::base_token_fetcher::BaseTokenFetcherConfig;

/// Trait used to query the stack's native token conversion rate. Used to properly
/// determine gas prices, as they partially depend on L1 gas prices, denominated in `eth`.
#[async_trait]
pub trait ConversionRateFetcher: 'static + std::fmt::Debug + Send + Sync {
fn conversion_rate(&self) -> anyhow::Result<u64>;
async fn update(&self) -> anyhow::Result<()>;
}

#[derive(Debug)]
pub(crate) struct NoOpConversionRateFetcher;

impl NoOpConversionRateFetcher {
pub fn new() -> Self {
Self
}
}

#[async_trait]
impl ConversionRateFetcher for NoOpConversionRateFetcher {
fn conversion_rate(&self) -> anyhow::Result<u64> {
Ok(1)
}

async fn update(&self) -> anyhow::Result<()> {
Ok(())
}
}

/// Struct in charge of periodically querying and caching the native token's conversion rate
/// to `eth`.
#[derive(Debug)]
pub(crate) struct BaseTokenFetcher {
pub config: BaseTokenFetcherConfig,
pub latest_to_eth_conversion_rate: AtomicU64,
http_client: reqwest::Client,
error_reporter: Arc<Mutex<ErrorReporter>>,
}

impl BaseTokenFetcher {
pub(crate) async fn new(config: BaseTokenFetcherConfig) -> anyhow::Result<Self> {
let http_client = reqwest::Client::new();

let conversion_rate = http_client
.get(format!(
"{}/conversion_rate/0x{}",
config.host,
config.token_address.encode_hex::<String>()
))
.send()
.await?
.json::<u64>()
.await
.context("Unable to parse the response of the native token conversion rate server")?;

let error_reporter = Arc::new(Mutex::new(ErrorReporter::new()));

Ok(Self {
config,
latest_to_eth_conversion_rate: AtomicU64::new(conversion_rate),
http_client,
error_reporter,
})
}
}

#[async_trait]
impl ConversionRateFetcher for BaseTokenFetcher {
fn conversion_rate(&self) -> anyhow::Result<u64> {
anyhow::Ok(
self.latest_to_eth_conversion_rate
.load(std::sync::atomic::Ordering::Relaxed),
)
}

async fn update(&self) -> anyhow::Result<()> {
match self
.http_client
.get(format!(
"{}/conversion_rate/0x{}",
&self.config.host,
&self.config.token_address.encode_hex::<String>()
))
.send()
.await
{
Ok(response) => {
let conversion_rate = response.json::<u64>().await.context(
"Unable to parse the response of the native token conversion rate server",
)?;
self.latest_to_eth_conversion_rate
.store(conversion_rate, std::sync::atomic::Ordering::Relaxed);
self.error_reporter.lock().await.reset();
}
Err(err) => self
.error_reporter
.lock()
.await
.process(anyhow::anyhow!(err)),
}

Ok(())
}
}

#[derive(Debug)]
struct ErrorReporter {
current_try: u8,
alert_spawned: bool,
}

impl ErrorReporter {
const MAX_CONSECUTIVE_NETWORK_ERRORS: u8 = 10;

fn new() -> Self {
Self {
current_try: 0,
alert_spawned: false,
}
}

fn reset(&mut self) {
self.current_try = 0;
self.alert_spawned = false;
}

fn process(&mut self, err: anyhow::Error) {
self.current_try = min(self.current_try + 1, Self::MAX_CONSECUTIVE_NETWORK_ERRORS);

tracing::error!("Failed to fetch native token conversion rate from the server: {err}");

if self.current_try >= Self::MAX_CONSECUTIVE_NETWORK_ERRORS && !self.alert_spawned {
vlog::capture_message(&err.to_string(), vlog::AlertLevel::Warning);
self.alert_spawned = true;
}
}
}
35 changes: 35 additions & 0 deletions core/lib/zksync_core/src/dev_api_conversion_rate/mod.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
use axum::{extract, extract::Json, routing::get, Router};
use tokio::sync::watch;
use zksync_config::configs::base_token_fetcher::BaseTokenFetcherConfig;

pub(crate) async fn run_server(
mut stop_receiver: watch::Receiver<bool>,
server_configs: &BaseTokenFetcherConfig,
) -> anyhow::Result<()> {
let app = Router::new().route("/conversion_rate/:token_address", get(get_conversion_rate));

let bind_address = if server_configs.host.starts_with("http://") {
&server_configs.host[7..] // If it starts with "http://", strip the prefix
} else {
&server_configs.host // Otherwise, return the original string
};

axum::Server::bind(&bind_address.parse().expect("Unable to parse socket address"))
.serve(app.into_make_service())
.with_graceful_shutdown(async move {
if stop_receiver.changed().await.is_err() {
tracing::warn!("Stop signal sender for conversion rate API server was dropped without sending a signal");
}
tracing::info!("Stop signal received, conversion rate server is shutting down");
})
.await
.expect("Conversion rate server failed");
tracing::info!("Conversion rate server shut down");
Ok(())
}

// basic handler that responds with a static string
async fn get_conversion_rate(extract::Path(_token_address): extract::Path<String>) -> Json<u64> {
tracing::info!("Received request for conversion rate");
Json(42)
}
17 changes: 14 additions & 3 deletions core/lib/zksync_core/src/l1_gas_price/gas_adjuster/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ use zksync_types::{U256, U64};

use self::metrics::METRICS;
use super::L1TxParamsProvider;
use crate::state_keeper::metrics::KEEPER_METRICS;
use crate::{base_token_fetcher::ConversionRateFetcher, state_keeper::metrics::KEEPER_METRICS};

mod metrics;
#[cfg(test)]
Expand All @@ -33,13 +33,15 @@ pub struct GasAdjuster {
pub(super) config: GasAdjusterConfig,
pubdata_sending_mode: PubdataSendingMode,
eth_client: Arc<dyn EthInterface>,
base_token_fetcher: Arc<dyn ConversionRateFetcher>,
}

impl GasAdjuster {
pub async fn new(
eth_client: Arc<dyn EthInterface>,
config: GasAdjusterConfig,
pubdata_sending_mode: PubdataSendingMode,
base_token_fetcher: Arc<dyn ConversionRateFetcher>,
) -> Result<Self, Error> {
// Subtracting 1 from the "latest" block number to prevent errors in case
// the info about the latest block is not yet present on the node.
Expand Down Expand Up @@ -70,6 +72,7 @@ impl GasAdjuster {
&last_block_blob_base_fee,
),
config,
base_token_fetcher,
pubdata_sending_mode,
eth_client,
})
Expand Down Expand Up @@ -157,6 +160,13 @@ impl GasAdjuster {
tracing::warn!("Cannot add the base fee to gas statistics: {}", err);
}

if let Err(err) = self.base_token_fetcher.update().await {
tracing::warn!(
"Error when trying to fetch the native erc20 conversion rate: {}",
err
);
}

tokio::time::sleep(self.config.poll_period()).await;
}
Ok(())
Expand All @@ -174,8 +184,9 @@ impl GasAdjuster {
let calculated_price =
(self.config.internal_l1_pricing_multiplier * effective_gas_price as f64) as u64;

// Bound the price if it's too high.
self.bound_gas_price(calculated_price)
let conversion_rate = self.base_token_fetcher.conversion_rate().unwrap_or(1);

self.bound_gas_price(calculated_price) * conversion_rate
}

pub(crate) fn estimate_effective_pubdata_price(&self) -> u64 {
Expand Down
Loading
Loading