Proposal on internal & external data in vocdoni-node

[arnaucube]

Proposal on internal & external data in vocdoni-node

Proposal that affects the vocdoni-node internals. This is an initial draft of the proposal, to start conversations regarding this topic and find a consensus for future iterations of the vocdoni-node.

Main idea

Current status is that vocdoni-node uses protobuf data structures both for external data transport and for internal logic of the node.
The main idea of this proposal is that internal structs should not need to be the same than the ones used for data transport. This means to use protobuf only for data transport and not for internal logic of the vocdoni-node.

So, protobuf data structs would be only used in the 'api/rpc layer' when processing the incoming data and when sending data outside, but from 'doors inside' the vocdoni-node would use its own defined internal structs.

As a 'rule of thumb', think of protobuf being an efficient replacement for json data packets, but internally in the node the jsons are converted into own internal structs, so the same would happen with the protobuf structs, they would be converted into internal structs of the node.

Additionally, protobuf structs could be used also to encode into bytes the data structs that are stored into key-value database.

Example of this approach used in Tendermint:

• InternalStructToProto: https://github.com/tendermint/tendermint/blob/eaa04681900efe928e0553a0c1f85385589072e1/types/block.go#L208
• ProtoToInternalStruct: https://github.com/tendermint/tendermint/blob/eaa04681900efe928e0553a0c1f85385589072e1/types/block.go#L231

Reasoning

Having a thin layer between the external-protobuf-structs and the internal-structs, has multiple benefits:

1. Data checks can be done when parsing the data from external-protobuf-struct into internal-struct
   • avoiding needing to add data correctness checks inside the vochain logic
     • this is how currently the checks are done, an some times the check is at multiple locations as it's embedded in other logic --> when possible, data checks should be outside from internal logic
   • finding errors when the data is entering the node, not after the incorrect data has been traveling across many internal functions
2. Endianness and other encoding confusions can be avoided:
   • the node can work internally in little-endian, and externally accept different endainness for different parameters
   • example:
     • imagine that there is an internal type that is a byte-array which represents a bigInt in little-endian, and there is anotherone which represents the bigInt in big-endian
     • if there is a superior reason to keep the code with both endianness, this can managed by this layer when doing the ProtoToInternalStruct
   • and furthermore, if at some point one endianness changes on client side, it just needs to be updated this thin layer, while the internal logic of the vocdoni-node can keep working as previously
3. Reduce development time of cycles between client & backend
   • we could have unit tests checking all the internal-struct to external-protobuf-struct and viceversa, so when integration flows are done nobody needs to spent time debugging which struct is giving problems
4. Reduce development time 2: Avoid needing to update protobuf to iterate internal logic of the vocdoni-node
   • currently if the internal logic of the vocdoni-node requires a change in a struct parameter needs to first update the dvote-protobuf repo and then import the new version into the vocdoni-node repo and do the logic changes
   • the internal logic could be agnostic to protobuf, so
5. Protobuf data structs have limitations: using the protobuf structs internally in the node can be tricky, there are cases where we need structs that contain an arrays of arrays, which is a type not straight-forward supported in protobuf (and to support it, it needs some 'tricks').

[mvdan]

@brickpop might have thoughts :)

I generally agree that public-facing API types should be separate. Right now they leak into many places where they shouldn't.

That said, I think this might be easier said than done: we interface with a bunch of different pieces at the data layer (database, vochain, APIs), and it's likely they won't all share the same types. So we might end up with one internal type, and 2+ external-facing types. We should be careful about unnecessary duplication.

[brickpop]

I agree with the thoughts exposed by @mvdan. Indeed, the protocol v2 proposal revolves around the idea that the Transport data models have their own definition, tightly aligned with the definition of the data that we are wrapping, signing, bundling, etc. Then, any components participating in the network, adapt to them, and not the other way around.

So, protobuf data structs would be only used in the 'api/rpc layer' when processing the incoming data and when sending data outside, but from 'doors inside' the vocdoni-node would use its own defined internal structs.

I totally agree that the api/rpc layer would have its own adapters to transform the incoming data into structs that the Domain layer of the Vochain can handle without dependencies (Clean code phylosophy main points). This doesn't prevent the Vochain from using its own protobuf serialization if deemed necessary, but the way things are handled internally shouldn't be leaking elsewhere, other than the Vochain itself.

To the benefits exposed by @arnaucube I would add that having a transport-level protobuf model:

• Would have a single source of truth (as opposed to docs, GW code, readme's, etc)
• Acting as a sucint documentation as well
• It's compiled. Errors arise immediately, as soon as an invalid field/type's are being used and not in the distant future.
• No more E2E-only testing.
  • Unit testing would be much easier if the Vochain modules only need to parse their relevant protobuf buffers, and don't need to spawn an entire deployment or even see flags/settings that belong to other domain boundaries.
  • Same for clients, they wouldn't need to rely on starting fake GW services to return forged data, if they were able to handle any protobuf data, regardless of its origin (Gateway, VocOne, WS, offline files, etc)
• Upgradeability of potentially static data

I would like to write an outline of the proto v2 proposal's main points, which align very much with your view, Arnau.

I also think that this is a change that would be rolled out incrementally. Deprecating certain of the current API's as soon as the Transport RPC models become available on https://server/proto