SVM Immutable Storage

[YaronWittenstein]

Immutable Storage

Overview

This SMIP outlines the required changes to SVM for introducing Immutable Storage.
This feature is a prerequisite for the Spawn implementation.

Goals and motivation

The current verify implementation assumes no access to the executing account's storage at all.
This restriction is very limiting regarding what can be achieved, and we'd like to lift it a bit.
However, we still want to keep the verify stage executing in a restricted mode, allowing read access to storage parts.

High-level design

Supporting the Immutable Storage will involve a couple of SVM breaking changes in these areas:

• Template layout
• Global State
• Runtime
• SDK

Template Layout Changes

Today each Template has storage layouts associated with it, along with the Wasm code.
However, in practice, only a single layout is emitted when using the current SVM SDK.

Right now, there is no notion of access mode attached to a layout.
Happily, the original layout design anticipated possible changes in SVM requirements that will demand different functions to execute in different modes.

In such a case, different Wasm functions could interact against the same layout, each one running under a different access mode.

Immutable Storage will require adding an explicit annotation as part of its layout definition.
When a layout is immutable, it needs to stay as such regardless of the code interacting with it.
The most naive and future extensible implementation will be adding a header for each layout.

Right now, having a flag stating whether the layout is immutable or not should suffice.
Lastly, moving the existing LayoutKind and having the Start Variable Id and End Variable Id to the layout header seems natural.

Back to the verify stage - it doesn't cost any Gas when it returns false.
Therefore, it must run very fast and have a VERIFY_GAS_CAPACITY assigned.
In addition to that, the preparation work before executing the verify Wasm code should be highly optimized.

Global-State Changes

The upcoming Spawn transaction changes will involve receiving the Immutable Storage as part of the transaction input.

Before executing the verify, the Global State needs to be informed of the-to-be-spawned Account Immutable Storage.

This execution path must be optimized and be independent of the given Immutable Storage layout. For example, whether it specifies, 2 or 100 variables should not matter.

To achieve that goal, the Global State needs to be aware of the Immutable Storage, at least to some extent.
As a result, the Global State should be more flexible in storing data.

We want to inform the Global State that a given blob of bytes represents a couple of fine-grained storage variables.
We want them to sit together under one physical key under the underlying key-value store.
In other words, the exact layout doesn't matter at all in that context.

The default behavior should be running verify as fast as we can without touching the disk.
It means that providing the Global State with an Immutable Storage blob associated with an account should keep it in memory by default.

Only when executing the Spawn constructor later, we'll make the Immutable Storage part of the dirty changes of the transaction. These changes will be committed later if everything turns out fine - i.e. when the Spawn transaction succeeded.

The Transaction Life-Cycle SMIP says that the verify code might execute multiple times throughout a transaction's lifetime. So that's another reason we'd want to have the Global State keep the Immutable Storage of an account in memory unless told otherwise.

In terms of the exact implementation, it'd probably be easiest to call clear on the Global State after executing the verify. The implications will be discarding any dirty changes - which are setting values to the Immutable Storage.

Runtime Changes

The Runtime is in charge of determining when access to a mutable storage variable is allowed and when not.
Host functions that interact against the account's storage will have to undergo validations.

If a host function is being asked to access mutable storage when it isn't allowed to - it should panic.
The only relevant scenario is the verify - but restricting access will likely be required for more cases. (i.e., the Nonces Abstractions).

Similarly, asking a host function to override an immutable variable should cause the program to panic.
As explained above, the abstraction introduced to the Global State should ask a blob of bytes (representing one or multiple storage variables) to be stored under one entry.

From the Global State perspective, there is no such thing as a mutable or immutable variable. Instead, the Runtime manages the capabilities of each function.

SVM SDK Changes

The SDK will have to be extended to support the Immutable Storage as well. One possible implementation will be adding an optional #[immutable] on top of each Field under the currently used
#[storage] Rust Struct. When not using this attribute annotation, we'll infer that the variable is a mutable one.
The code generation part of the SDK will group the immutable variables to reside under the same Immutable Storage layout, and the rest variables will belong to the default Layout of mutable variables.
In addition to that, the emitted JSONs (upon successful compilation) will need to be modified to report whether a variable is immutable or not.

Codec Changes

As for the codec, the Start Variable Id exists under the Data Section of a Template.
The codec will have to be modified to reflect the new changes of the Template storage.
Saying that each Layout will consist of a Header and Variables Sizes seems a good solution (for both svm-layout and svm-codec crates).

Questions/concerns

• Limiting the total blob size is essential since a Template Immutable Storage will occupy a single key under the scenes.
  It's not supposed to be a big issue, but it's a point to be taken into account. Technically, a Template should manage with multiple Immutable and Mutable layouts.

• This SMIP adds a bit of complexity to the Global State implementation since it needs to support two different translations mechanisms.

One will be the currently existing one, and the second will be mapping immutable variables.
One that will address Immutable Variables and one for the Mutable Variables (as exists today).
Maybe a Router on top of the existing Global State code could simplify things a bit.

Dependencies and interactions

• As detailed, implementing the Immutable Storage SMIP entails touching many components of SVM.
• The SVM API (and go-svm) are supposed to stay intact.

Stakeholders and reviewers

@noamnelke
@lrettig
@neysofu
@avive
@moshababo

[neysofu]

Blocker for the changes to verify that we need as documented by spacemeshos/pm#108.