Publishing get stuck/delay if RabbitMQ is unresponsve

[dgonzalezKentech]

Describe the bug
We believe in some weird situation when rabbitMQ server is overloaded or network issue (we are still investigating this part), the client get stuck when publishing until it is resolved (not clear yet what resolved it).
Still investigated what was the situation but it isn't the first time.
One other case we had 8M message in the queue and after a quick inject of another 660K (which were Nack immediately but the consumer client), we get the same situation "stuck" situation with our Producer.

The error we get after the time the lock is "fixed" is the channel closed:
[err:Exception (504) Reason: "channel/connection is not open"]

To Reproduce
Still investigating what exactly happened, we only know for some reason publishing is getting stuck (one main culprit so far, logs are not helping yet)
github.com/rabbitmq/amqp091-go v1.10.0
github.com/wagslane/go-rabbitmq v0.14.2

Expected behavior
When calling publish without confirmation, to be like an async function, just to not care what happened to the message, even if the server is not there.
We could accept to just wait a little bit, controller, with timeout (context), for publishing, but current

Additional context
Service running in docker, publishing around 500msg/s all the time.
RabbitMQ cluster - 2 nodes. (version 3.3.5)

We are aware we are using the Wrapper go-rabbitmq, but after checking carefully, we don't see a specific place where it could get lock, except some flow controls which we have NOT enabled.

Publishing with options:

if err := r.publisher.PublishWithContext(
		ctx,
		data,
		[]string{routingKey},
		rabbitmq.WithPublishOptionsContentType(_contentTypeJSON),
		rabbitmq.WithPublishOptionsHeaders(headers),
		rabbitmq.WithPublishOptionsMandatory,
		rabbitmq.WithPublishOptionsPersistentDelivery,
		rabbitmq.WithPublishOptionsExchange(messageSetup.Exchange),
	); err != nil {
		return fmt.Errorf("[RabbitRepo][Publish][exchange:%s][routingKey:%s][err:%w]", messageSetup.Exchange, routingKey, err)
	}

Publisher:

rabbitmq.NewPublisher(
		conn,
		rabbitmq.WithPublisherOptionsLogger(config.RabbitMQ().Logger),
		rabbitmq.WithPublisherOptionsExchangeName(config.RabbitMQ().ExchangeNotification),
	)

where default options are:

ExchangeOptions: ExchangeOptions{
			Name:       "",
			Kind:       amqp.ExchangeDirect,
			Durable:    false,
			AutoDelete: false,
			Internal:   false,
			NoWait:     false,
			Passive:    false,
			Args:       Table{},
			Declare:    false,
		},
		Logger:      stdDebugLogger{},
		ConfirmMode: false,

Honestly, the most time I spend writing/documenting this issue the more I think the problem is with the reconnect mechanism of the other library (go-rabbitmq), because I see reconnections messages.....
But just to focus my attention in the other library and design the proper test strategy......

Could you please confirm if possible if there is any situation where publishing could get stuck? with the amp091-go library I mean....
Like if the server is not there or the channel doesn't know if the server is there, or it is not properly closed, or waiting to be closed.
Or instead if it is virtually impossible that calling "Publish" from amp091-go the response is not immediately. Again, not taking about "PublishWithDeferredConfirm"

Thanks.

[lukebakken]

RabbitMQ cluster - 2 nodes. (version 3.3.5)

Two serious issues here:

• 2-node clusters are neither recommended nor supported. You must use an odd number of nodes.
• RabbitMQ 3.3.5 is extremely old, so I'm assuming you mean 3.13.5

Your report is basically asking us to guess what might be happening, and we don't have the time for that.

The only way we will be able to assist you is if you come up with code we can run with specific reproduction steps to see this issue.

[dgonzalezKentech]

Hi,

to begin with, thanks for the quick response,

I am investigating the case further, where I will do some Load Test.
True my report is quite long but basically I am just wondering if this library, there is any stage where it gets waiting for the server.
I understand this question might sounds odd... because if you knew that already I am sure there would be a solution or a proper configuration. (Just trying to get as much information as possible, but I understand time is limited for every body).

(And not, this is a production system with STILL running 3.3.5.... we are planning the migration.

In any case, thanks again, If I manage to get a reproducible scenario and get some debugging information, will revert back with the notes.

[Zerpet]

Reading about cluster alarms, specifically about memory alarms may help you in your investigation. Specially this phrase may be relevant:

By default, including if the limit hint is not configured, a RabbitMQ node will use about 60% of the available RAM, it raises a memory alarm and will block all connections that are publishing messages

This library does not have any back-pressure or blocking mechanism built-in.

[dgonzalezKentech]

Thank you. Will ask the DevOps onsite to take a look.

First step I am pushing is to work in the migration (upgrade). We are running a too old version.