Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(runtime) - Mitigate the receipt size limit bug #12633

Merged
merged 16 commits into from
Jan 8, 2025
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions runtime/runtime/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ use std::cmp::max;
use std::collections::{HashMap, HashSet, VecDeque};
use std::sync::Arc;
use tracing::{debug, instrument};
use verifier::ValidateReceiptMode;

mod actions;
pub mod adapter;
Expand Down Expand Up @@ -670,6 +671,7 @@ impl Runtime {
&apply_state.config.wasm_config.limit_config,
receipt,
apply_state.current_protocol_version,
ValidateReceiptMode::NewReceipt,
)
}) {
new_result.result = Err(ActionErrorKind::NewReceiptValidationError(e).into());
Expand Down Expand Up @@ -1868,6 +1870,7 @@ impl Runtime {
&processing_state.apply_state.config.wasm_config.limit_config,
&receipt,
protocol_version,
ValidateReceiptMode::OldReceipt,
)
.map_err(|e| {
StorageError::StorageInconsistentState(format!(
Expand Down Expand Up @@ -1933,6 +1936,7 @@ impl Runtime {
&processing_state.apply_state.config.wasm_config.limit_config,
receipt,
protocol_version,
ValidateReceiptMode::OldReceipt,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, maybe it would be better to name this variant something along the lines of StoredReceipt or something like that? My intuition is that for delayed receipts "old" is perhaps still applicable, sure, but incoming receipts don't feel like they would qualify for the "old" qualifier.

At the same time I'm trying to wrap around if it is really alright to not validate the incoming receipts for their size? I guess not doing so is the reason why this PR doesn't actually fix the associated issue fully?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, maybe it would be better to name this variant something along the lines of StoredReceipt or something like that? My intuition is that for delayed receipts "old" is perhaps still applicable, sure, but incoming receipts don't feel like they would qualify for the "old" qualifier.

Stored doesn't really fit for incoming receipts. Maybe "ExistingReceipt"? The distinction is between receipts that were already created and validated and totally new ones

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the same time I'm trying to wrap around if it is really alright to not validate the incoming receipts for their size? I guess not doing so is the reason why this PR doesn't actually fix the associated issue fully?

I think it's fine because all incoming receipts were created by runtime, which we trust fully. We can trust that the runtime doesn't create receipts that are too big, invalid, etc.
In this case it turned out that it's possible to bypass runtime's size check, and because of that the runtime produced invalid receipts. Fixing that would require adding proper max_receipt_size checks when applying an action receipt. This is a bit complicated, so for now we have a mitigation.

)
.map_err(RuntimeError::ReceiptValidationError)?;
if processing_state.total.compute >= compute_limit
Expand Down
33 changes: 26 additions & 7 deletions runtime/runtime/src/verifier.rs
Original file line number Diff line number Diff line change
Expand Up @@ -293,14 +293,17 @@ pub(crate) fn validate_receipt(
limit_config: &LimitConfig,
receipt: &Receipt,
current_protocol_version: ProtocolVersion,
mode: ValidateReceiptMode,
) -> Result<(), ReceiptValidationError> {
let receipt_size: u64 =
borsh::to_vec(receipt).unwrap().len().try_into().expect("Can't convert usize to u64");
if receipt_size > limit_config.max_receipt_size {
return Err(ReceiptValidationError::ReceiptSizeExceeded {
size: receipt_size,
limit: limit_config.max_receipt_size,
});
if mode == ValidateReceiptMode::NewReceipt {
let receipt_size: u64 =
borsh::to_vec(receipt).unwrap().len().try_into().expect("Can't convert usize to u64");
if receipt_size > limit_config.max_receipt_size {
return Err(ReceiptValidationError::ReceiptSizeExceeded {
size: receipt_size,
limit: limit_config.max_receipt_size,
});
}
}

// We retain these checks here as to maintain backwards compatibility
Expand All @@ -325,6 +328,21 @@ pub(crate) fn validate_receipt(
}
}

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ValidateReceiptMode {
/// Used for validating new receipts that were just created.
/// More strict than `OldReceipt` mode, which has to handle older receipts.
NewReceipt,
/// Used for validating older receipts that were saved in the state/received. Less strict than
/// NewReceipt validation. Tolerates some receipts that wouldn't pass new validation. It has to
/// be less strict because:
/// 1) Older receipts might have been created before new validation rules.
/// 2) There is a bug which allows to create receipts that are above the size limit. Runtime has
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought it's not possible to create them above the size limit but that the receipt may increase in size later?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here I used "create" in a broader sense, I wanted to say that it's possible for a large receipt to occur.
And when a max size value is returned, the created receipt is larger than max_receipt_size, so I'd say that counts as "create"

/// to handle them gracefully until the receipt size limit bug is fixed.
/// See https://github.com/near/nearcore/issues/12606 for details.
OldReceipt,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. I can see that the delayed and incoming receipts are validated but the buffered receipts are not, as far as I can see. Is it an omission?
  2. Is the validation for old receipts actually necessary or is it just an extra protection? Those receipts are a result of on chain computation so I would assume we trust them here already.

Copy link
Contributor Author

@jancionear jancionear Jan 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think validating old receipts is just a sanity check. When an old receipt fails validation the runtime panics, so we don't expect this validation to ever fail. I think it's fine to not validate buffered receipts.

}

/// Validates given ActionReceipt. Checks validity of the number of input data dependencies and all actions.
fn validate_action_receipt(
limit_config: &LimitConfig,
Expand Down Expand Up @@ -1540,6 +1558,7 @@ mod tests {
&limit_config,
&Receipt::new_balance_refund(&alice_account(), 10, ReceiptPriority::NoPriority),
PROTOCOL_VERSION,
ValidateReceiptMode::NewReceipt,
)
.expect("valid receipt");
}
Expand Down