Feature/ensembles wait for riak kv #649
Conversation
You can now choose to save the intercept modules in basho-patches so that they are loaded on a restart. This should be useful to modify Riak's behavior at startup time.
Now ensemble peers are prevented from starting up until the riak_kv service is up to avoid nasty races that could even lead to node crashes as the ensembles frantically query for data that isn't ready.
| lager:info("Verifying peers wait for riak_kv_service"), | ||
| rt_intercept:add_and_save(Node, {riak_kv_vnode, [{{init, 1}, {[], | ||
| fun(Args) -> | ||
| timer:sleep(5000), |
There was a problem hiding this comment.
Might be a good idea to make the sleep duration configurable.
|
Test failed twice when using the appropriate kv and ensemble branches. Test output: https://gist.github.com/lordnull/ca2ee6d88eb0e1fcb095 |
|
@lordnull I realize how this can fail. I didn't do the check the way I thought it in my head. Will be pushing a fix soon. We need to take the list of ensembles first, then check for the service. If the service is off, assert on the list if it's not empty. The way it is now, it is possible for the peers to start because the riak_kv service has started since we checked. |
Changing to fetching the list of peers first, then check if the riak_kv service is up. If the service is up, then check the peers. Otherwise it is possible to see the service down, then peers up because it went up in the interim. Also, making KV vnode delay configurable.
|
The last commit should address the check race and makes the delay configurable @lordnull |
|
Test fails w/o the ensemble/kv patch, passes with them, so it's a 👍 from me. |
|
@lordnull does the 👍 extend to the PRs? basho/riak_kv#989 basho/riak_ensemble#32 |
Feature/ensembles wait for riak kv
Test that ensemble peers will not start until the riak_kv service is up. This verifies basho/riak_kv#989 and basho/riak_ensemble#32
Functionality was added to the intercepts so that they can save the generated modules in the basho-patches directory of a node, so that they can be loaded on startup. This should be useful to other tests that require modifying the behavior of Riak while it's starting up. /cc @jburwell since you needed something like this recently.