Command committed without majority (on ra_machine crash)

[visciang]

Describe the bug

If a ra_machine crashes while processing a command, the command is committed on the raft log even if no majority is reached on the cluster.

1. setup a cluster of 2 nodes (node1, node2) where we simulate a specific node crash on the follower (node2) via explicit raise
2. issue a process_command on the leader (node1)
3. the ra_machine on the follower restarts and terminate but the process_command succeeds and is committed

Note:
I know that ra machines must be written not to crash #294 (comment), but in this case I expect the command not to be committed.

Reproduction steps

iex --name [email protected] -S mix
iex --name [email protected] -S mix

#--- on node1:

nodes = [:"[email protected]", :"[email protected]"]
server_ids = for node <- nodes, do: {:quick_start, node}
for node <- nodes, do: true = Node.connect(node)
for node <- nodes, do: :rpc.call(node, :ra, :start, [])

cluster_name = :quick_start
machine = {:simple,
  fn command, _state ->
    if node() == :"[email protected]" do  # << SIMULATE A FAULT (ie out of memory in the erlang VM)
      raise "ERROR"
    end
    command
  end,
0}

{:ok, servers_started, _servers_not_started} = :ra.start_cluster(:default, cluster_name, machine, server_ids)
leader = :ra_leaderboard.lookup_leader(cluster_name)

# on the leader :"[email protected]"
:ra.process_command(leader, "COMMAND")

ra_server on node2 restarts and then the supervisor terminates it (as expected)
but on node1 process_command returns {:ok, "COMMAND", {:quick_start, :"[email protected]"}}.

Expected behavior

Since the issued command is not acknowledged by node2 the cluster has no agreed on the log entry.
Therefore the command should not be committed.

Additional context

No response

[michaelklishin]

Two node clusters are an explicitly unsupported configuration: there is no such thing as a majority when one node fails.

[kjnilsson]

This is expected behaviour. The leader doesn't care if the follower was able to process the command it only cares about if the command was added to the follower's log successfully, which it was (if it wasn't in this case the follower would never have tried to apply it).

Please ensure your state machine logic is deterministic and does not rely on environment specific data points.

[visciang]

Got it! Thank you