Repair after replication conflict/split brain

[Totktonada]

Background

There are situations, when a node in a replicaset ends up with conflicting history of operations. I know the following cases:

• A node performs transactions itself (it is in the read-write mode) and receives a replication stream. In the master-master mode or if several nodes becomes writeable by a mistake a node can observe conflicting transactions: one originating from the node and one from the replication stream, accessing the same data at the same time. At this point the replication stream is considered broken and one of nodes should be excluded from the replicaset.
• After tarantool/tarantool@af7d703 a replicaset with raft-based leader election can detect the situation when a node attempts to connect to the replicaset after split-brain. The node can't connect to the replicaset disregarding, whether there were transactions accessing the same data.

Asynchronous transactions can conflict, that's normal. The leader election quorum can be descreased for the sake of the service availability, at least to continue to write to some spaces. It is valid situation too.

What solutions we offer now:

• Think what to do at conflict beforehand, add before_replace trigger¹ and, of course, write tests for this logic.
• Rebootstrap the node with loss of the new data on it.

The first way has its own complexities and is not always possible. The second way also has disadvantages:

• Reboostrap is not a fast process and if something goes wrong during it (say, another conflict), we're in a double trouble. It also worth to note that the difference in data likely is tiny comparing the whole data set and it seems counter-intuitive to re-fetch all data.
• We lost the new data, while there are cases, when we could merge them with interation with a human.

Proposal

I propose to implement a tool for repairing from such situations. The flow is very similar to resolving conflicts in git.

The tool will look at two history of operations, detect the conflict point and dump the difference into a human readable file. After this a human can resolve all conflicts: choose one or another variant of data or merge them somehow. The resulting file can be applied on the healthy part of the replicaset. The history of operations of the broken node can be rewinded to the conflict point.

I guess it is also possible to interact only with the broken node: rewind it to the conflict point, apply all new changes from the healthy part of the replicaset, apply conflict resolution changes afterwards. After this it should be possible to connect this node to the rest of the replicaset.

Materials

[1] https://www.tarantool.io/en/doc/latest/book/replication/repl_problem_solving/
[2] https://stackoverflow.com/questions/56734280/conflict-resolution-in-tarantool-how-to-fix-replication-in-master-master-mode-i
[3] tarantool/tarantool@af7d703

Footnotes

1. Using the on_commit trigger inside the on_replace trigger on _space system space, which should be set in the on_schema_init trigger. ↩