Tests for removing nodes without breaking riak_ensemble

[andrewjstone]

Tests for basho/riak_core#572

[andrewjstone]

Requires:

basho/riak_ensemble#19
basho/riak_kv#926
basho/riak_core#578

#570

[lordnull]

This needs a rebase; one of the tests references rt_intercept_pt which exists on master but not on this branch.

Running a rebased version causes a failure still, though:

11:26:08.225 [warning] ensemble_remove_node failed: {{assertEqual_failed,[{module,ensemble_remove_node},{line,70},{expression,"length ( Cluster )"},{expected,1},{value,0}]},[{ensemble_remove_node,'-confirm/0-fun-7-',2,[{file,"tests/ensemble_remove_node.erl"},{line,70}]},{ensemble_remove_node,confirm,0,[{file,"tests/ensemble_remove_node.erl"},{line,70}]},{riak_test_runner,return_to_exit,3,[{file,"src/riak_test_runner.erl"},{line,159}]}]}
11:26:08.225 [error] Error in process <0.190.0> on node '[email protected]' with exit value: {{assertEqual_failed,[{module,ensemble_remove_node},{line,70},{expression,"length ( Cluster )"},{expected,1},{value,0}]},[{ensemble_remove_node,'-confirm/0-fun-7-',2,[{file,"tests/ensemble_remove_node.erl"},{line,70}]},{ensemble_remove_node... 


11:26:08.225 [error] 
================ ensemble_remove_node failure stack trace =====================
{{assertEqual_failed,[{module,ensemble_remove_node},
                      {line,70},
                      {expression,"length ( Cluster )"},
                      {expected,1},
                      {value,0}]},
 [{ensemble_remove_node,'-confirm/0-fun-7-',2,
                        [{file,"tests/ensemble_remove_node.erl"},{line,70}]},
  {ensemble_remove_node,confirm,0,
                        [{file,"tests/ensemble_remove_node.erl"},{line,70}]},
  {riak_test_runner,return_to_exit,3,
                    [{file,"src/riak_test_runner.erl"},{line,159}]}]}
===============================================================================

[jtuple]

This pull-request adds two new tests: ensemble_remove_node and ensemble_remove_node2.

As desired, ensemble_remove_node fails against develop without the related pull-requests applied; and passes with the pull-requests applied.

ensemble_remove_node2 is a bit tricker. It tests a condition that should only be true after merging the necessary pull-requests. Yet, it still passes against develop. Why? Well, it turns out there is an unrelated bug that can hit a race condition which mirrors the condition the test is checking. As luck would have it, this bug is indirectly fixed by the changes in the associated pull-requests as well.

So, I've extended the test slightly in f822e52 to explicitly verify that the intercept used in the test is the reason why nodes are not exiting, rather than from the unrelated bug that previously existed on master. As extended, the test now fails against develop, but passes after having the necessary pull-requests applied.

So, +1 on these tests. Although, please double check my changes to the test.

Once the related pull-requests are reviewed/merged, these tests can be merged in. No need to rebase as @lordnull suggested as this pull-request merges cleanly and work just fine after merging.

[jtuple]

Finished reviewing complete feature change across all pull-requests. +1 to merge basho/riak_ensemble#19, basho/riak_core#578, and basho/riak_kv#926

FYI, looks like I forgot to assign myself to this issue when I first started this review ~2 weeks ago. Sorry for the duplicate effort @lordnull, but thanks for the extra pair of eyes.

[andrewjstone]

Test extension looks good @jtuple
👍