- #PXC-347: Non-PRIM during membership change

[sysprg]

Timers in gcomm (which implemented using ASIO library)
can be triggered even during normal shutdown sequence -
even after interruption of the main event loop in gcomm
subsystem and completion of the corresponding thread.

For example, during normal shutdown sequence and after
termination of the main event loop in gcomm subsystem and
even after completion of the corresponding thread by the
close() function (from the gcs_gcomm.cpp file), the
following call path still possible:

handle_timers -> handle_inactivity_timer -> check_inactive()

Function handle_install_timer() also may called from the
"gcomm::evs::Proto::handle_timers()" (from evs_proto.cpp
module).

Also, gcomm::GMCast::reconnect() function (from gmcast.cpp
file) may be called from the gcomm::GMCast::handle_timers().

Thus, we have a 100% accurate evidence that after the
termination of the main gcomm message loop (using the
close() method from the gcs_gcomm.cpp file), we still
continue to handle events associated with the timers.

Perhaps this is not a direct bug of the server, but an
unexpected feature of the implementation of the timers
in ASIO, which allows posing the events associated with
timers even after an interruption of the main event loop
(using "terminate" function).

For example, we see the following lines in the "epoll_reactor::run(...)"
method from the epoll_reactor.ipp file, which is used in most of our
test systems:

    if (ptr == &interrupter_)
    {
      // No need to reset the interrupter since we're leaving the descriptor
      // in a ready-to-read state and relying on edge-triggered notifications
      // to make it so that we only get woken up when the descriptor's epoll
      // registration is updated.

#if defined(ASIO_HAS_TIMERFD)
      if (timer_fd_ == -1)
	check_timers = true;
#else // defined(ASIO_HAS_TIMERFD)
      check_timers = true;
#endif // defined(ASIO_HAS_TIMERFD)
    }
#if defined(ASIO_HAS_TIMERFD)
    else if (ptr == &timer_fd_)
    {
 ...
     check_timers = true;
    }
#endif // defined(ASIO_HAS_TIMERFD){code}

As we can see, even after interruption of the main loop through
the "interrupter_" handler, the ASIO code sets the "check_timers"
variable to "true", and then it can add all the events associated
with the timers to the queue:

  if (check_timers)
  {
    mutex::scoped_lock common_lock(mutex_);
    timer_queues_.get_ready_timers(ops);
...

Later, the timer handlers, which called after the termination
of main loop during normal shutdown sequence, can form false
"view", in which all the nodes are declared as "partitioned".

I think that it is sending this "view" over the network, and
this leads to the transfer of entire cluster into NON-PRIMARY
state.

Therefore, to fix this, I propose to make two changes:

1. We need to stop responding to the timers after termination
   of the main gcomm-related event loop (by the close() function
   from the gcs_gcomm.cpp file).

2. I offer completely disable support of the timerfd feature
   in ASIO, which has already proved to be a source of error.
   I have already told about one of these problems with ASIO
   developers (by publishing PR in their GitHub) and I even
   got a positive reaction to the pull request in their GitHub,
   but yet there is no official release of the ASIO, which
   fixes all the problems related to timerfd feature.

Shutting down the timerfd feature does not affect the
performance of real applications - I was not able to fix
the difference in the tests.

Unfortunately, I cannot attach TC to the patch, because
the only way to make TC for it (that I came up with) requires
DEBUG_SYNC, but during normal shutdown sequence, it stops
working, as management of the DEBUG_SYNC in MTR is actually
carried out through conventional SQL operators, and the
system is already in the process of shutdown.