[Deliverable] Store v3 - store synchronisation

[chair28980]

Epics

• [Epic: research] Store v3 - store synchronisation research#104
• [Epic: nwaku] Store v3 - store synchronisation nwaku#3071
• [Epic: QA] Store v3 - store synchronisation #226

Summary

• Goal: After this upgrade, the network will provide distributed and synchronised store services.
• Deliverables: (2), (3)
• Research tracks: Message Reliability
• Estimated date of completion: 2024Q2

An improved version of the Store protocol, marking a crucial increment towards a synchronisation protocol:

• introduces the concept of deterministic message hashes to index messages
• considers the Store as a key-value store
• allows for querying a list of keys (message hashes) from the Store
• allows for querying for the full message content (values) or a set of keys from the Store
• keeps all previous value-based filtering (e.g. content topic, timestamp) in place

The proposed PR to simplify the Store protocol and use message
hashes as index/cursor, can be used as a starting point.

Store v3 - store synchronisation

• Goal: Upgrade the Store service capability in the network from a collection of local, unsynchronised,
  semi-centralised (trusted) service nodes to a decentralised service capability in the network with inter-node synchronisation.
• Deliverables: (2), (3)
• Research tracks: Message Reliability
• Estimated date of completion: 2024Q2

Building on Store v3-beta, this version of Store includes basic synchronisation between nodes.
This will probably include:

• a protocol/heuristic to resume store services after an offline period
• a protocol/heuristic to periodically compare local key-value store with other nodes and find missing keys
• a protocol/heuristic to periodically download the messages (values) for missing keys from other store nodes

Note that this can either be

• (i) designing a new synchronisation protocol built into the Store protocol
• (ii) adapting existing synchronisation building blocks (e.g. a Prolly Tree library) into Store, or
• (iii) simply swapping the key-value store itself to a synchronised backend, such as GossipLog

IMO (iii) should be pursued as the preferred option, as far as possible.

[SionoiS]

Tasks are mostly independent of each other and can be tackled in any order.

• Research: RFC and specs.

  • List the assumptions we are making
  • Define the wire messages
  • Refer to negentropy for how it works

• Reseach & Nwaku: New Nwaku protocol feat: Nwaku Sync nwaku#2403

  • This can be done without the bindings for the negentropy lib, just mock the payloads.
  • Code the interactions between 2 peers.
  • Define how other parts of the code will interact with Waku Sync

• Nwaku: Nim bindings for negentropy

  • Should be easy to do since we are exchanging mostly bytes through the language bridge.
  • Nim will not manage memory in this case. All the hashes will be sent across since storage will be on the C++ side.
  • Might be easier to update the rust crate and wrap it instead of C++ (I never used C++ send help! :P)

• Nwaku: Hooking negentropy to a DB

  • They use LMDB we don't. How hard will it be to switch to postgres is unknown.
  • Could we switch to LMDB instead of postgres? Or maybe use both?

[SionoiS]

@chair28980 work has been scoped for research and nwaku. Not sure about the other teams.

[Ivansete-status]

Great summary @SionoiS !

May elaborate more this point plz?

Nwaku: Nim bindings for negentropy

---

Re the following, we will need to switch to another solution that is more convenient for Waku and therefore I see the green light for investigating other databases. Postgres is aimed to work for Status and we can maintain both for now.

Could we switch to LMDB instead of postgres? Or maybe use both?

[SionoiS]

May elaborate more this point plz?

Nwaku: Nim bindings for negentropy

Negentropy is the C++ lib that implement RBSR and to use it we'll need nim bindings.

Re the following, we will need to switch to another solution that is more convenient for Waku and therefore I see the green light for investigating other databases. Postgres is aimed to work for Status and we can maintain both for now.

Could we switch to LMDB instead of postgres? Or maybe use both?

My guess is that the easiest thing to do is to use LMDB for SYNC and keep everything else unchanged.