Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: don't crash client if stopping an unknown consumer, stop brod_consumer processes when terminating brod_group_subscriber_v2 #601

Merged
merged 3 commits into from
Oct 25, 2024

Conversation

thalesmg
Copy link
Contributor

@thalesmg thalesmg commented Oct 25, 2024

Crash prior to fix

=ERROR REPORT==== 25-Oct-2024::12:11:26.244821 ===
** Generic server t_double_stop_consumer terminating
** Last message in was {stop_consumer,<<"brod-client-SUITE-topic">>}
** When Server state == {state,t_double_stop_consumer,
                               [{"localhost",9192}],
                               <0.664.0>,[],<0.667.0>,<0.668.0>,
                               [{get_metadata_timeout_seconds,10}],
                               t_double_stop_consumer}
** Reason for termination ==
** {{badmatch,{error,not_found}},
    [{brod_client,handle_call,3,
                  [{file,"/home/thales/dev/emqx/brod/src/brod_client.erl"},
                   {line,391}]},
     {gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,1131}]},
     {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,1160}]},
     {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,241}]}]}
** Client <0.661.0> stacktrace
** [{gen,do_call,4,[{file,"gen.erl"},{line,240}]},
    {gen_server,call,3,[{file,"gen_server.erl"},{line,415}]},
    {brod_client,safe_gen_call,3,
                 [{file,"/home/thales/dev/emqx/brod/src/brod_client.erl"},
                  {line,979}]},
    {brod_client_SUITE,t_double_stop_consumer,1,
                       [{file,"/home/thales/dev/emqx/brod/test/brod_client_SUITE.erl"},
                        {line,395}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
    {test_server,run_test_case_eval1,6,[{file,"test_server.erl"},{line,1302}]},
    {test_server,run_test_case_eval,9,[{file,"test_server.erl"},{line,1234}]}]

=CRASH REPORT==== 25-Oct-2024::12:11:26.244863 ===
  crasher:
    initial call: brod_client:init/1
    pid: <0.663.0>
    registered_name: t_double_stop_consumer
    exception error: no match of right hand side value {error,not_found}
      in function  brod_client:handle_call/3 (/home/thales/dev/emqx/brod/src/brod_client.erl, line 391)
      in call from gen_server:try_handle_call/4 (gen_server.erl, line 1131)
      in call from gen_server:handle_msg/6 (gen_server.erl, line 1160)
    ancestors: [brod_sup,<0.657.0>]
    message_queue_len: 3
    messages: [{'EXIT',<0.667.0>,shutdown},
                  {'EXIT',<0.668.0>,shutdown},
                  {'EXIT',<0.664.0>,shutdown}]
    links: [<0.658.0>]
    dictionary: [{rand_seed,{#{type => exsss,next => #Fun<rand.0.65977474>,
                                bits => 58,uniform => #Fun<rand.1.65977474>,
                                uniform_n => #Fun<rand.2.65977474>,
                                jump => #Fun<rand.3.65977474>},
                              [52087848521744273|168058237035269794]}}]
    trap_exit: true
    status: running
    heap_size: 1598
    stack_size: 28
    reductions: 2571
  neighbours:

=ERROR REPORT==== 25-Oct-2024::12:11:26.244935 ===
    supervisor: {local,brod_sup}
    errorContext: child_terminated
    reason: {{badmatch,{error,not_found}},
             [{brod_client,handle_call,3,
                           [{file,"/home/thales/dev/emqx/brod/src/brod_client.erl"},
                            {line,391}]},
              {gen_server,try_handle_call,4,
                          [{file,"gen_server.erl"},{line,1131}]},
              {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,1160}]},
              {proc_lib,init_p_do_apply,3,
                        [{file,"proc_lib.erl"},{line,241}]}]}
    offender: [{pid,<0.663.0>},
               {id,t_double_stop_consumer},
               {mfargs,{brod_client,start_link,
                                    [[{"localhost",9192}],
                                     t_double_stop_consumer,
                                     [{get_metadata_timeout_seconds,10}]]}},
               {restart_type,{permanent,10}},
               {shutdown,5000},
               {child_type,worker}]

Termination of brod_consumer processes

Previously, terminating a brod_group_subscriber_v2 would leave brod_consumer processes started by brod_client lingering. Now, we cleanup those processes upon termination.

```
=ERROR REPORT==== 25-Oct-2024::12:11:26.244821 ===
** Generic server t_double_stop_consumer terminating
** Last message in was {stop_consumer,<<"brod-client-SUITE-topic">>}
** When Server state == {state,t_double_stop_consumer,
                               [{"localhost",9192}],
                               <0.664.0>,[],<0.667.0>,<0.668.0>,
                               [{get_metadata_timeout_seconds,10}],
                               t_double_stop_consumer}
** Reason for termination ==
** {{badmatch,{error,not_found}},
    [{brod_client,handle_call,3,
                  [{file,"/home/thales/dev/emqx/brod/src/brod_client.erl"},
                   {line,391}]},
     {gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,1131}]},
     {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,1160}]},
     {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,241}]}]}
** Client <0.661.0> stacktrace
** [{gen,do_call,4,[{file,"gen.erl"},{line,240}]},
    {gen_server,call,3,[{file,"gen_server.erl"},{line,415}]},
    {brod_client,safe_gen_call,3,
                 [{file,"/home/thales/dev/emqx/brod/src/brod_client.erl"},
                  {line,979}]},
    {brod_client_SUITE,t_double_stop_consumer,1,
                       [{file,"/home/thales/dev/emqx/brod/test/brod_client_SUITE.erl"},
                        {line,395}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
    {test_server,run_test_case_eval1,6,[{file,"test_server.erl"},{line,1302}]},
    {test_server,run_test_case_eval,9,[{file,"test_server.erl"},{line,1234}]}]

=CRASH REPORT==== 25-Oct-2024::12:11:26.244863 ===
  crasher:
    initial call: brod_client:init/1
    pid: <0.663.0>
    registered_name: t_double_stop_consumer
    exception error: no match of right hand side value {error,not_found}
      in function  brod_client:handle_call/3 (/home/thales/dev/emqx/brod/src/brod_client.erl, line 391)
      in call from gen_server:try_handle_call/4 (gen_server.erl, line 1131)
      in call from gen_server:handle_msg/6 (gen_server.erl, line 1160)
    ancestors: [brod_sup,<0.657.0>]
    message_queue_len: 3
    messages: [{'EXIT',<0.667.0>,shutdown},
                  {'EXIT',<0.668.0>,shutdown},
                  {'EXIT',<0.664.0>,shutdown}]
    links: [<0.658.0>]
    dictionary: [{rand_seed,{#{type => exsss,next => #Fun<rand.0.65977474>,
                                bits => 58,uniform => #Fun<rand.1.65977474>,
                                uniform_n => #Fun<rand.2.65977474>,
                                jump => #Fun<rand.3.65977474>},
                              [52087848521744273|168058237035269794]}}]
    trap_exit: true
    status: running
    heap_size: 1598
    stack_size: 28
    reductions: 2571
  neighbours:

=ERROR REPORT==== 25-Oct-2024::12:11:26.244935 ===
    supervisor: {local,brod_sup}
    errorContext: child_terminated
    reason: {{badmatch,{error,not_found}},
             [{brod_client,handle_call,3,
                           [{file,"/home/thales/dev/emqx/brod/src/brod_client.erl"},
                            {line,391}]},
              {gen_server,try_handle_call,4,
                          [{file,"gen_server.erl"},{line,1131}]},
              {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,1160}]},
              {proc_lib,init_p_do_apply,3,
                        [{file,"proc_lib.erl"},{line,241}]}]}
    offender: [{pid,<0.663.0>},
               {id,t_double_stop_consumer},
               {mfargs,{brod_client,start_link,
                                    [[{"localhost",9192}],
                                     t_double_stop_consumer,
                                     [{get_metadata_timeout_seconds,10}]]}},
               {restart_type,{permanent,10}},
               {shutdown,5000},
               {child_type,worker}]
```
@thalesmg thalesmg changed the title fix: don't crash if stopping an unknown consumer fix: don't crash client if stopping an unknown consumer, stop brod_consumer processes when terminating brod_group_subscriber_v2 Oct 25, 2024
@thalesmg thalesmg marked this pull request as ready for review October 25, 2024 17:00
@thalesmg thalesmg merged commit 3e44cd4 into kafka4beam:master Oct 25, 2024
14 checks passed
@thalesmg thalesmg deleted the 20241025-fix-stop-consumer-crash branch October 25, 2024 18:02
thalesmg added a commit to zmstone/emqx that referenced this pull request Oct 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants