RabbitMQ 3.13.2 on Erlang 26.2.3 logs an exception "timeout,{gen_server,call,[os_mon_sysinfo,get_mem_info]" on Windows with CrowdStrike tool suite

[sharad578]

Describe the bug

RabbitMQ 3.13.2 with Erlang 26.2.3 is terminating with timeout,{gen_server,call,[os_mon_sysinfo,get_mem_info] reason in Windows host. It is also creating erlang core dump at time of shutdown. Why does RabbitMQ is giving following Error

RabbitMQ log

2024-09-10 22:01:41.845000-05:00 [error] <0.128.0> ** Generic server memsup terminating
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0> ** Last message in was {'EXIT',<0.48421.0>,
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0>                            {timeout,
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0>                                {gen_server,call,
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0>                                    [os_mon_sysinfo,get_mem_info]}}}
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0> ** When Server state == [{data,[{"Timeout",60000}]},
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0>                          {items,{"Memory Usage",
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0>                                  [{"Allocated",85208141824},
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0>                                   {"Total",549651329024}]}},
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0>                          {items,{"Worst Memory User",
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0>                                  [{"Pid",<0.722.0>},{"Memory",14662216}]}}]
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0> ** Reason for termination ==
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0> ** {timeout,{gen_server,call,[os_mon_sysinfo,get_mem_info]}}
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0> 
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0>   crasher:
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0>     initial call: memsup:init/1
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0>     pid: <0.128.0>
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0>     registered_name: memsup
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0>     exception exit: {timeout,{gen_server,call,[os_mon_sysinfo,get_mem_info]}}
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0>       in function  gen_server:handle_common_reply/8 (gen_server.erl, line 1226)
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0>     ancestors: [os_mon_sup,<0.124.0>]
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0>     message_queue_len: 0
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0>     messages: []
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0>     links: [<0.125.0>]
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0>     dictionary: []
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0>     trap_exit: true
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0>     status: running
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0>     heap_size: 6772
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0>     stack_size: 28
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0>     reductions: 70921755
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0>   neighbours:
2024-09-10 22:01:41.845000-05:00 [error] <0.128.0> 
2024-09-10 22:01:41.845000-05:00 [error] <0.125.0>     supervisor: {local,os_mon_sup}
2024-09-10 22:01:41.845000-05:00 [error] <0.125.0>     errorContext: child_terminated
2024-09-10 22:01:41.845000-05:00 [error] <0.125.0>     reason: {timeout,{gen_server,call,[os_mon_sysinfo,get_mem_info]}}
2024-09-10 22:01:41.845000-05:00 [error] <0.125.0>     offender: [{pid,<0.128.0>},
2024-09-10 22:01:41.845000-05:00 [error] <0.125.0>                {id,memsup},
2024-09-10 22:01:41.845000-05:00 [error] <0.125.0>                {mfargs,{memsup,start_link,[]}},
2024-09-10 22:01:41.845000-05:00 [error] <0.125.0>                {restart_type,permanent},
2024-09-10 22:01:41.845000-05:00 [error] <0.125.0>                {significant,false},
2024-09-10 22:01:41.845000-05:00 [error] <0.125.0>                {shutdown,2000},
2024-09-10 22:01:41.845000-05:00 [error] <0.125.0>                {child_type,worker}]
2024-09-10 22:01:41.845000-05:00 [error] <0.125.0> 
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0> ** Generic server memsup terminating
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0> ** Last message in was {'EXIT',<0.48429.0>,
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0>                            {timeout,
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0>                                {gen_server,call,
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0>                                    [os_mon_sysinfo,get_mem_info]}}}
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0> ** When Server state == "memsup:format_status/2 crashed"
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0> ** Reason for termination ==
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0> ** {timeout,{gen_server,call,[os_mon_sysinfo,get_mem_info]}}
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0> 
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0>   crasher:
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0>     initial call: memsup:init/1
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0>     pid: <0.48428.0>
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0>     registered_name: memsup
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0>     exception exit: {timeout,{gen_server,call,[os_mon_sysinfo,get_mem_info]}}
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0>       in function  gen_server:handle_common_reply/8 (gen_server.erl, line 1226)
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0>     ancestors: [os_mon_sup,<0.124.0>]
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0>     message_queue_len: 0
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0>     messages: []
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0>     links: [<0.125.0>]
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0>     dictionary: []
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0>     trap_exit: true
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0>     status: running
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0>     heap_size: 4185
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0>     stack_size: 28
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0>     reductions: 7734
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0>   neighbours:
2024-09-10 22:01:46.846000-05:00 [error] <0.48428.0> 
2024-09-10 22:01:46.846000-05:00 [error] <0.125.0>     supervisor: {local,os_mon_sup}
2024-09-10 22:01:46.846000-05:00 [error] <0.125.0>     errorContext: child_terminated
2024-09-10 22:01:46.846000-05:00 [error] <0.125.0>     reason: {timeout,{gen_server,call,[os_mon_sysinfo,get_mem_info]}}
2024-09-10 22:01:46.846000-05:00 [error] <0.125.0>     offender: [{pid,<0.48428.0>},
2024-09-10 22:01:46.846000-05:00 [error] <0.125.0>                {id,memsup},
2024-09-10 22:01:46.846000-05:00 [error] <0.125.0>                {mfargs,{memsup,start_link,[]}},
2024-09-10 22:01:46.846000-05:00 [error] <0.125.0>                {restart_type,permanent},
2024-09-10 22:01:46.846000-05:00 [error] <0.125.0>                {significant,false},
2024-09-10 22:01:46.846000-05:00 [error] <0.125.0>                {shutdown,2000},
2024-09-10 22:01:46.846000-05:00 [error] <0.125.0>                {child_type,worker}]
2024-09-10 22:01:46.846000-05:00 [error] <0.125.0> 
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0> ** Generic server memsup terminating
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0> ** Last message in was {'EXIT',<0.48431.0>,
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0>                            {timeout,
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0>                                {gen_server,call,
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0>                                    [os_mon_sysinfo,get_mem_info]}}}
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0> ** When Server state == "memsup:format_status/2 crashed"
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0> ** Reason for termination ==
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0> ** {timeout,{gen_server,call,[os_mon_sysinfo,get_mem_info]}}
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0> 
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0>   crasher:
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0>     initial call: memsup:init/1
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0>     pid: <0.48430.0>
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0>     registered_name: memsup
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0>     exception exit: {timeout,{gen_server,call,[os_mon_sysinfo,get_mem_info]}}
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0>       in function  gen_server:handle_common_reply/8 (gen_server.erl, line 1226)
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0>     ancestors: [os_mon_sup,<0.124.0>]
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0>     message_queue_len: 0
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0>     messages: []
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0>     links: [<0.125.0>]
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0>     dictionary: []
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0>     trap_exit: true
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0>     status: running
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0>     heap_size: 4185
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0>     stack_size: 28
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0>     reductions: 7734
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0>   neighbours:
2024-09-10 22:01:51.849000-05:00 [error] <0.48430.0> 
2024-09-10 22:01:51.849000-05:00 [error] <0.125.0>     supervisor: {local,os_mon_sup}
2024-09-10 22:01:51.849000-05:00 [error] <0.125.0>     errorContext: child_terminated
2024-09-10 22:01:51.849000-05:00 [error] <0.125.0>     reason: {timeout,{gen_server,call,[os_mon_sysinfo,get_mem_info]}}
2024-09-10 22:01:51.849000-05:00 [error] <0.125.0>     offender: [{pid,<0.48430.0>},
2024-09-10 22:01:51.849000-05:00 [error] <0.125.0>                {id,memsup},
2024-09-10 22:01:51.849000-05:00 [error] <0.125.0>                {mfargs,{memsup,start_link,[]}},
2024-09-10 22:01:51.849000-05:00 [error] <0.125.0>                {restart_type,permanent},
2024-09-10 22:01:51.849000-05:00 [error] <0.125.0>                {significant,false},
2024-09-10 22:01:51.849000-05:00 [error] <0.125.0>                {shutdown,2000},
2024-09-10 22:01:51.849000-05:00 [error] <0.125.0>                {child_type,worker}]
2024-09-10 22:01:51.849000-05:00 [error] <0.125.0> 
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0> ** Generic server memsup terminating
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0> ** Last message in was {'EXIT',<0.48433.0>,
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0>                            {timeout,
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0>                                {gen_server,call,
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0>                                    [os_mon_sysinfo,get_mem_info]}}}
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0> ** When Server state == "memsup:format_status/2 crashed"
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0> ** Reason for termination ==
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0> ** {timeout,{gen_server,call,[os_mon_sysinfo,get_mem_info]}}
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0> 
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0>   crasher:
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0>     initial call: memsup:init/1
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0>     pid: <0.48432.0>
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0>     registered_name: memsup
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0>     exception exit: {timeout,{gen_server,call,[os_mon_sysinfo,get_mem_info]}}
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0>       in function  gen_server:handle_common_reply/8 (gen_server.erl, line 1226)
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0>     ancestors: [os_mon_sup,<0.124.0>]
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0>     message_queue_len: 0
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0>     messages: []
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0>     links: [<0.125.0>]
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0>     dictionary: []
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0>     trap_exit: true
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0>     status: running
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0>     heap_size: 4185
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0>     stack_size: 28
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0>     reductions: 7734
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0>   neighbours:
2024-09-10 22:01:56.851000-05:00 [error] <0.48432.0> 
2024-09-10 22:01:56.851000-05:00 [error] <0.125.0>     supervisor: {local,os_mon_sup}
2024-09-10 22:01:56.851000-05:00 [error] <0.125.0>     errorContext: child_terminated
2024-09-10 22:01:56.851000-05:00 [error] <0.125.0>     reason: {timeout,{gen_server,call,[os_mon_sysinfo,get_mem_info]}}
2024-09-10 22:01:56.851000-05:00 [error] <0.125.0>     offender: [{pid,<0.48432.0>},
2024-09-10 22:01:56.851000-05:00 [error] <0.125.0>                {id,memsup},
2024-09-10 22:01:56.851000-05:00 [error] <0.125.0>                {mfargs,{memsup,start_link,[]}},
2024-09-10 22:01:56.851000-05:00 [error] <0.125.0>                {restart_type,permanent},
2024-09-10 22:01:56.851000-05:00 [error] <0.125.0>                {significant,false},
2024-09-10 22:01:56.851000-05:00 [error] <0.125.0>                {shutdown,2000},
2024-09-10 22:01:56.851000-05:00 [error] <0.125.0>                {child_type,worker}]
2024-09-10 22:01:56.851000-05:00 [error] <0.125.0> 
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0> ** Generic server memsup terminating
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0> ** Last message in was {'EXIT',<0.48435.0>,
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0>                            {timeout,
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0>                                {gen_server,call,
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0>                                    [os_mon_sysinfo,get_mem_info]}}}
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0> ** When Server state == "memsup:format_status/2 crashed"
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0> ** Reason for termination ==
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0> ** {timeout,{gen_server,call,[os_mon_sysinfo,get_mem_info]}}
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0> 
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0>   crasher:
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0>     initial call: memsup:init/1
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0>     pid: <0.48434.0>
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0>     registered_name: memsup
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0>     exception exit: {timeout,{gen_server,call,[os_mon_sysinfo,get_mem_info]}}
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0>       in function  gen_server:handle_common_reply/8 (gen_server.erl, line 1226)
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0>     ancestors: [os_mon_sup,<0.124.0>]
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0>     message_queue_len: 0
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0>     messages: []
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0>     links: [<0.125.0>]
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0>     dictionary: []
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0>     trap_exit: true
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0>     status: running
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0>     heap_size: 4185
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0>     stack_size: 28
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0>     reductions: 7734
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0>   neighbours:
2024-09-10 22:02:01.852000-05:00 [error] <0.48434.0> 
2024-09-10 22:02:01.852000-05:00 [error] <0.125.0>     supervisor: {local,os_mon_sup}
2024-09-10 22:02:01.852000-05:00 [error] <0.125.0>     errorContext: child_terminated
2024-09-10 22:02:01.852000-05:00 [error] <0.125.0>     reason: {timeout,{gen_server,call,[os_mon_sysinfo,get_mem_info]}}
2024-09-10 22:02:01.852000-05:00 [error] <0.125.0>     offender: [{pid,<0.48434.0>},
2024-09-10 22:02:01.852000-05:00 [error] <0.125.0>                {id,memsup},
2024-09-10 22:02:01.852000-05:00 [error] <0.125.0>                {mfargs,{memsup,start_link,[]}},
2024-09-10 22:02:01.852000-05:00 [error] <0.125.0>                {restart_type,permanent},
2024-09-10 22:02:01.852000-05:00 [error] <0.125.0>                {significant,false},
2024-09-10 22:02:01.852000-05:00 [error] <0.125.0>                {shutdown,2000},
2024-09-10 22:02:01.852000-05:00 [error] <0.125.0>                {child_type,worker}]
2024-09-10 22:02:01.852000-05:00 [error] <0.125.0> 
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0> ** Generic server memsup terminating
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0> ** Last message in was {'EXIT',<0.48437.0>,
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0>                            {timeout,
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0>                                {gen_server,call,
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0>                                    [os_mon_sysinfo,get_mem_info]}}}
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0> ** When Server state == "memsup:format_status/2 crashed"
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0> ** Reason for termination ==
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0> ** {timeout,{gen_server,call,[os_mon_sysinfo,get_mem_info]}}
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0> 
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0>   crasher:
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0>     initial call: memsup:init/1
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0>     pid: <0.48436.0>
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0>     registered_name: memsup
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0>     exception exit: {timeout,{gen_server,call,[os_mon_sysinfo,get_mem_info]}}
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0>       in function  gen_server:handle_common_reply/8 (gen_server.erl, line 1226)
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0>     ancestors: [os_mon_sup,<0.124.0>]
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0>     message_queue_len: 0
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0>     messages: []
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0>     links: [<0.125.0>]
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0>     dictionary: []
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0>     trap_exit: true
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0>     status: running
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0>     heap_size: 4185
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0>     stack_size: 28
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0>     reductions: 7734
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0>   neighbours:
2024-09-10 22:02:06.854000-05:00 [error] <0.48436.0> 
2024-09-10 22:02:06.854000-05:00 [error] <0.125.0>     supervisor: {local,os_mon_sup}
2024-09-10 22:02:06.854000-05:00 [error] <0.125.0>     errorContext: child_terminated
2024-09-10 22:02:06.854000-05:00 [error] <0.125.0>     reason: {timeout,{gen_server,call,[os_mon_sysinfo,get_mem_info]}}
2024-09-10 22:02:06.854000-05:00 [error] <0.125.0>     offender: [{pid,<0.48436.0>},
2024-09-10 22:02:06.854000-05:00 [error] <0.125.0>                {id,memsup},
2024-09-10 22:02:06.854000-05:00 [error] <0.125.0>                {mfargs,{memsup,start_link,[]}},
2024-09-10 22:02:06.854000-05:00 [error] <0.125.0>                {restart_type,permanent},
2024-09-10 22:02:06.854000-05:00 [error] <0.125.0>                {significant,false},
2024-09-10 22:02:06.854000-05:00 [error] <0.125.0>                {shutdown,2000},
2024-09-10 22:02:06.854000-05:00 [error] <0.125.0>                {child_type,worker}]
2024-09-10 22:02:06.854000-05:00 [error] <0.125.0> 
2024-09-10 22:02:06.854000-05:00 [error] <0.125.0>     supervisor: {local,os_mon_sup}
2024-09-10 22:02:06.854000-05:00 [error] <0.125.0>     errorContext: shutdown
2024-09-10 22:02:06.854000-05:00 [error] <0.125.0>     reason: reached_max_restart_intensity
2024-09-10 22:02:06.854000-05:00 [error] <0.125.0>     offender: [{pid,<0.48436.0>},
2024-09-10 22:02:06.854000-05:00 [error] <0.125.0>                {id,memsup},
2024-09-10 22:02:06.854000-05:00 [error] <0.125.0>                {mfargs,{memsup,start_link,[]}},
2024-09-10 22:02:06.854000-05:00 [error] <0.125.0>                {restart_type,permanent},
2024-09-10 22:02:06.854000-05:00 [error] <0.125.0>                {significant,false},
2024-09-10 22:02:06.854000-05:00 [error] <0.125.0>                {shutdown,2000},
2024-09-10 22:02:06.854000-05:00 [error] <0.125.0>                {child_type,worker}]
2024-09-10 22:02:06.854000-05:00 [error] <0.125.0> 
2024-09-10 22:02:08.856000-05:00 [error] <0.125.0>     supervisor: {local,os_mon_sup}
2024-09-10 22:02:08.856000-05:00 [error] <0.125.0>     errorContext: shutdown_error
2024-09-10 22:02:08.856000-05:00 [error] <0.125.0>     reason: killed
2024-09-10 22:02:08.856000-05:00 [error] <0.125.0>     offender: [{pid,<0.126.0>},
2024-09-10 22:02:08.856000-05:00 [error] <0.125.0>                {id,os_mon_sysinfo},
2024-09-10 22:02:08.856000-05:00 [error] <0.125.0>                {mfargs,{os_mon_sysinfo,start_link,[]}},
2024-09-10 22:02:08.856000-05:00 [error] <0.125.0>                {restart_type,permanent},
2024-09-10 22:02:08.856000-05:00 [error] <0.125.0>                {significant,false},
2024-09-10 22:02:08.856000-05:00 [error] <0.125.0>                {shutdown,2000},
2024-09-10 22:02:08.856000-05:00 [error] <0.125.0>                {child_type,worker}]
2024-09-10 22:02:08.856000-05:00 [error] <0.125.0> 
2024-09-10 22:02:08.856000-05:00 [warning] <0.403.0> Free disk space monitoring failed to retrieve the amount of available space: {killed,
2024-09-10 22:02:08.856000-05:00 [warning] <0.403.0>                                                                               {gen_server,
2024-09-10 22:02:08.856000-05:00 [warning] <0.403.0>                                                                                call,
2024-09-10 22:02:08.856000-05:00 [warning] <0.403.0>                                                                                [os_mon_sysinfo,
2024-09-10 22:02:08.856000-05:00 [warning] <0.403.0>                                                                                 get_disk_info]}}
2024-09-10 22:02:08.863000-05:00 [notice] <0.44.0> Application os_mon exited with reason: shutdown

Erlang coredump log

=erl_crash_dump:0.5
Tue Sep 10 22:02:21 2024
Slogan: Kernel pid terminated (application_controller) ("{application_terminated,os_mon,shutdown}")
System version: Erlang/OTP 26 [erts-14.2.3] [source] [64-bit] [smp:40:32] [ds:40:32:10] [async-threads:1]
Taints: dyntrace,pubkey_os_cacerts,asn1rt_nif,crypto
Atoms: 51025
Calling Thread: scheduler:10
=scheduler:1
Scheduler Sleep Info Flags: SLEEPING | TSE_SLEEPING | WAITING
Scheduler Sleep Info Aux Work: 
Current Port: 
Run Queue Max Length: 0
Run Queue High Length: 0
Run Queue Normal Length: 0
Run Queue Low Length: 0
Run Queue Port Length: 0
Run Queue Flags: OUT_OF_WORK | HALFTIME_OUT_OF_WORK
Current Process: 

Reproduction steps

1. This issue occur in Windows cluster.

[lukebakken]

Please answer the following questions:

• How long has this environment operated WITHOUT this issue? When did the issue start? What changed at that time?
• Do you have ANY anti-virus or security scanners in place? This includes Windows Defender.
• What does the Windows Event viewer log at the same time as these issues? It will be in the "System" or "Application" logs, most likely.

[sharad578]

RabbitMQ is running in this system for long time without giving this issue. 2 days back it starts giving the issue.
There is Crowd strike software is running in our system. We did not find any Scan related logs in System and Application log.

[lukebakken]

RabbitMQ is running in this system for long time without giving this issue. 2 days back it starts giving the issue.

What changed two days ago?

Please see @michaelklishin's comment here and follow his suggestions.

[michaelklishin]

CrowdStrike product suite does include a form of anti-virus that can prevent processes such as RabbitMQ from opening files, network connections, and other essential operations.

When running on Windows, RabbitMQ and Erlang processes and executables must be whitelisted/excluded from scanning by such tools.

We will not troubleshoot CrowdStrike tools for you. Certainly not for non-paying and non-contributing users. Please take it from here.

[sharad578]

Is there any rabbitMQ configurable parameter available to increase default timeout of 5 seconds at time of retrieving system information mentioned in #11358

[lukebakken]

No, and that probably wouldn't help. 5 seconds is already an eternity, so if it takes that long, it is indicative of something wrong with your environment.

[michaelklishin]

This is a duplicate of #11358, even the patch version is exactly the same.

RabbitMQ does not "crash", it is the runtime memory monitor that runs into an exception. So the effects of this should be one (obviously important but still) metric unavailable.

[michaelklishin]

@sharad578 please upgrade to 3.13.7 and disable/whitelist RabbitMQ and its data directory from all anti-virus software that may run on the host. If the node has only 1 or 2 cores, increase it to 4. CPU resources starvation is an unlikely but possible cause with timeouts like this, although OS process permissions limitations are more likely.

[sharad578]

We have 4 cores in our system. Is there any other way to fix this issue without upgrading to 3.13.7. Can we fix it through configuring some RabbitMQ configuration parameter. please suggest.

[sharad578]

this is the "rabbitmqctl status" log does not provide information about Free disk space, Connection count, Queue count, Virtual host count at time of failure. Our system have sufficient disk space but Free disk space shows "not available". why is it showing "not available" ?

"rabbitmqctl status" output is given below

Memory

Total memory used: 0.2227 gb
Calculation strategy: rss
Memory high watermark setting: 2.048 gb, computed to: 2.048 gb

allocated_unused: 0.0743 gb (33.38 %)
other_proc: 0.0432 gb (19.42 %)
other_system: 0.0429 gb (19.28 %)
code: 0.0358 gb (16.09 %)
binary: 0.0085 gb (3.83 %)
reserved_unallocated: 0.0042 gb (1.88 %)
metrics: 0.0038 gb (1.73 %)
plugins: 0.0033 gb (1.47 %)
atom: 0.0016 gb (0.7 %)
mgmt_db: 0.0011 gb (0.47 %)
msg_index: 0.001 gb (0.46 %)
connection_other: 0.001 gb (0.44 %)
connection_readers: 0.001 gb (0.43 %)
other_ets: 0.0003 gb (0.13 %)
queue_procs: 0.0002 gb (0.08 %)
metadata_store: 0.0002 gb (0.08 %)
mnesia: 0.0001 gb (0.05 %)
connection_channels: 0.0001 gb (0.03 %)
metadata_store_ets: 0.0 gb (0.01 %)
quorum_ets: 0.0 gb (0.01 %)
connection_writers: 0.0 gb (0.01 %)
quorum_queue_procs: 0.0 gb (0.0 %)
quorum_queue_dlx_procs: 0.0 gb (0.0 %)
stream_queue_procs: 0.0 gb (0.0 %)
stream_queue_replica_reader_procs: 0.0 gb (0.0 %)
queue_slave_procs: 0.0 gb (0.0 %)
stream_queue_coordinator_procs: 0.0 gb (0.0 %)

File Descriptors

Total: 20, limit: 65439
Sockets: 16, limit: 58893

Free Disk Space

Low free disk space watermark: 0.05 gb
Free disk space: (not available)

Totals

Connection count: 
Queue count: 
Virtual host count: 

[michaelklishin]

This is not expected or common to see. This node cannot collect some of the standard metrics. I do not recall this being reported elsewhere.

See logs for clues, and not just RabbitMQ logs.

[michaelklishin]

From another discussion on exactly the same topic.

os_mon_sysinfo:get_mem_info/1 is the monitor of available memory used on Windows. Most likely the node is prevented from fetching relevant information from the OS due to security policies or anti-virus software of sorts.

The exception is entirely harmless.

You can "disable" it by telling the node to use an absolute value instead:

# rabbitmq.conf or a conf.d file such as conf.d/total_memory_override.conf
total_memory_available_override_value = 4GB

Then the node won't try to infer that value. This is a rarely used feature that was added as a last resort measure for environment when a node cannot compute how much memory it has because years ago, Erlang's memory allocators were not cgroups-aware.