Unhandled QQ terminations via gen_statem:stop/3

[Ayanda-D]

Community Support Policy

• I have read RabbitMQ's Community Support Policy

RabbitMQ version used

4.0.3

How is RabbitMQ deployed?

Debian package

Steps to reproduce the behavior in question

Hi Team

I'm running make run-broker (or start-cluster) and creating a quorum queue, then manually terminating it via gen_statem:stop/3. The process terminates, but, the queue itself (from the metadata and monitoring tools) appears available (and marked live in state and running on the Mgmt-UI). I understand this is termination on the raft library components, but I expect the QQ to be entirely unavailable. What is the expected behaviour of the QQ after such a termination?

> make run-broker

>  rabbit_amqqueue:list().
[{amqqueue,{resource,<<"/">>,queue,<<"Q.1">>},
           true,false,none,
           [{<<"x-queue-type">>,longstr,<<"quorum">>}],
           {'%2F_Q.1',rabbit@host},
           [],[],[],undefined,undefined,[],[],live,0,[],<<"/">>,
           #{user => <<"guest">>},
           rabbit_quorum_queue,
           #{nodes => [rabbit@host]}}]**

> gen_statem:stop('%2F_Q.1', {shutdown, delete}, 5_000).
ok

> whereis('%2F_Q.1').
undefined

Queue still marked as live.

> rabbit_amqqueue:list().
[{amqqueue,{resource,<<"/">>,queue,<<"Q.1">>},
           true,false,none,
           [{<<"x-queue-type">>,longstr,<<"quorum">>}],
           {'%2F_Q.1',rabbit@host},
           [],[],[],undefined,undefined,[],[],live,0,[],<<"/">>,
           #{user => <<"guest">>},
           rabbit_quorum_queue,
           #{nodes => [rabbit@host]}}]

NOTE: We're still facing issues where QQ's have leaders unavailable ( failing with noproc ), but go unnoticed and re-election doesnt transfer leadership to one of the followers. Discussed #12701 and still rely on a custom CLI check to detect these. We want such terminations of the FSM^^^ to be propagated to all components of the QQ, e.g. including updating the metadata store (and preferably always guarantee a re-election is triggered, which doesnt seem to always be the case).

[michaelklishin]

@Ayanda-D you are welcome to submit pull requests with the changes you need. In particular since you have a way to reproduce.