Serialization format

[hackaugusto]

Serialization format discussion

Main points on our discussion:

• compact. This rules out self-describing format like JSON/YAML/BJSON
• interruption of deserialization

I think we should have two points in this discussion:

1. Should we use a standardize format
2. Should we use a library for serialization

These are orthogonal to one another, i.e. we can define our own data format and use a serialization/deserialization library or we can do neither.

standardard format

Pros:

• Better support for other ecosystems (assuming they have libraries for the given format)
• Takes advantage of existing encoding techniques, e.g. variable-length integer encoding/tightly packing of data, and format features, e.g. canonization.

Cons:

• Have to conform to the format, which may not be tailored to our use case

Example formats:

• very-compact: messagepack (Miden probably won't benefit from this though, since the compact representation are for common value, which probably are not that common here, i.e. small integers)
• compact: bare, borsh
• zero copy: flatbuffers, cap'n'proto'

library for serialization

Pros:

• Ensures we follow the format encoding rules
• Built-in checks

Cons:

• Lost control on encoding / decoding
• Additional dependency to evaluate / add to the compile time

Libraries

nom:

Pros:

• should be fast/safe (I haven't tested it)

Cons:

• it's only for deserialization

serde:

Pros:

• out-of-the-box support for multiple formats

Cons:

• it is not possible to control the serialization/deserialization process
• must fit data into the serde data model

Some benchmarks:

• https://ruststack.org/rust-serialization/
• https://github.com/djkoloski/rust_serialization_benchmark

[vlopes11]

I think its best if we prioritize a mechanism to allow customization/interruption of the parsing since this will be a critical point to avoid attack vectors.

AFAIK, we don't have a performant, safe way to inject user exceptions into any of these libraries, suggesting the deserialization might have to be done either manually or based on some generic framework.

As for serialization, considering this wouldn't be critical, we can go with any performant, reliable, and maintained option such as borsh. It uses only hashbrown as dependency, and this is a big plus for this option as we don't need to worry too much about increasing our stack risk by adding third-party deps.

Alternatively, we can implement our own serialization/deserialization with nom, if we are up to sacrifice additional effort to manually implement everything. This is analogous to not using any library at all (as we might just end reading bytes and parsing field elements), but nom provides a heavily maintained, reliable and reputable library.

[hackaugusto]

https://github.com/0xPolygonMiden/miden-node is using tonic, which is built on top of protobuf