Consensus mismatch in peer to peer application

[b-garbacz]

Background

I was working on a test application that involved multiple independent nodes. Each node was designed to broadcast X messages once all nodes were connected to the specified channel. Afterward, I planned to "export" the list of messages from each node and compare them to evaluate the system's "consensus".

For the project, I was using Corestore, Autobase and Hyperswarm.

The source code of each individual node can be found here.

Expected Result

After running an iteration with 10 nodes, each broadcasting 1,000 messages, I exported the data from each peer into a text file. The data consisted of a list of all the messages, arranged in the order in which they were received. I repeated the benchmark twice, resulting in a total of 20,000 messages in the P2P network.

Note: I had to stop the second benchmark midway due to unrelated issues, which means the number of messages in the exported lists from each node may differ, particularly for those with an index greater than the indexedLength of their original peer.

I expected that all the text files would contain the same messages in a consistent order, at least for those messages with index smaller than each view's indexedLength attribute.

Actual Result

In the end, the exported message lists from all peers (w1, w2, w3, ...) matched, with one exception: w10.

In this particular peer, the exported list showed a single difference when compared to the others: the first and second messages were swapped.

When I checked the indexedLength of w10, it was around 17,500, so messages 0 and 1 should have followed the same pattern as seen in the other peers, based on my understanding.

The raw Corestore data from each peer, along with the exported message lists from each node, can be downloaded here. The list corresponding to w10 is a94bc1302e8eb10368320c9955bd86880a7a9512942373a91cbdd7a80b61d7b4.

---

5 first messages in w10:

0 -  node: cd891517e5e8950c8ffba4fec343f16dc0b288067bd376dcc1f69fd42e594eed, idx: 0, content: ms to wait = 608 timestamp: 2025-01-09T14:02:02.420Z
1 -  node: 159bc6d72911b48ea07d778d142e1f3876d2c4c35f86748f9d9b13b93fa06990, idx: 0, content: ms to wait = 103, timestamp: 2025-01-09T14:02:02.256Z
2 -  node: 24ff6a1fbd74879fe02f49ee3f852e8a22f8cecc43ba66173bd6fd9dc2e6b4f3, idx: 0, content: ms to wait = 127, timestamp: 2025-01-09T14:02:02.312Z
3 -  node: 159bc6d72911b48ea07d778d142e1f3876d2c4c35f86748f9d9b13b93fa06990, idx: 1, content: ms to wait = 241, timestamp: 2025-01-09T14:02:02.505Z
4 -  node: 24ff6b941ea02d0151cf36cefb5b8f72eeac8badcdf013b9dff1b9ccf10d4842, idx: 0, content: ms to wait = 681, timestamp: 2025-01-09T14:02:02.234Z

5 first messages in other peers:

0 -  node: 159bc6d72911b48ea07d778d142e1f3876d2c4c35f86748f9d9b13b93fa06990, idx: 0, content: ms to wait = 103, timestamp: 2025-01-09T14:02:02.256Z
1 -  node: cd891517e5e8950c8ffba4fec343f16dc0b288067bd376dcc1f69fd42e594eed, idx: 0, content: ms to wait = 608, timestamp: 2025-01-09T14:02:02.420Z
2 -  node: 24ff6a1fbd74879fe02f49ee3f852e8a22f8cecc43ba66173bd6fd9dc2e6b4f3, idx: 0, content: ms to wait = 127, timestamp: 2025-01-09T14:02:02.312Z
3 -  node: 159bc6d72911b48ea07d778d142e1f3876d2c4c35f86748f9d9b13b93fa06990, idx: 1, content: ms to wait = 241, timestamp: 2025-01-09T14:02:02.505Z
4 -  node: 24ff6b941ea02d0151cf36cefb5b8f72eeac8badcdf013b9dff1b9ccf10d4842, idx: 0, content: ms to wait = 681, timestamp: 2025-01-09T14:02:02.234Z
5 -  node: 24ff6a1fbd74879fe02f49ee3f852e8a22f8cecc43ba66173bd6fd9dc2e6b4f3, idx: 0, content: ms to wait = 898, timestamp: 2025-01-09T14:02:02.061Z

[b-garbacz]

After a conversation with @mafintosh on Keet, to make it easier for you to investigate the issue, I've provided a link to download the complete program, can be downloaded here.

How the program works:

The program operates as follows:

1. The nodes begin sending messages between each other once the number of active nodes equals the global variable MIN_NODES.
2. Each node will broadcast a total of MESSAGE_COUNT messages within the network.
3. Use the command npx pear run . <wallet-name> to start a node.

Available commands

While the nodes are listening, the following commands can be executed in the command line:

• exit - terminates the node,
• list - displays the complete view of the node,
• export - exports the view of the node to a file.

If the global variable SAVE_WRITERS_TO_FILE is set to true, the names of the writers will be saved to a writers file. By default, this option is set to false.

The wallets mentioned in the issue, along with their corresponding writers, are already included in the project provided in the link above.