Persistent messages folder

[mathewvp]

Hi,

When writing persistent messages using this library, the messages are being written to "queues" folder but while using php amqp library, it is being written to "msg_store_persistent" folder. Where is this configuration being set? This is not an issue with the library but I need help to figure this out, thanks.

[lukebakken]

Could you please provide a minimal set of code I can run in each language to observe what you report?

Also, always let us know the following information when reporting an issue:

• Erlang and RabbitMQ version
• Operating systems and version
• Go version and Go client library version
• PHP version and php-amqplib version

[lukebakken]

I see you've also opened this issue - php-amqplib/php-amqplib#962

Please, we can't assist without a way to reproduce. What you're asking us to do now is write code and guess what you are doing. Since we're providing free support, we can't assist if you don't help us out a bit. Thanks for understanding.

[mathewvp]

Php code

 $exchange = new \AMQPExchange($this-_channel);
 $exchange-setName('xxx.push-notification.exg');
 $exchange-setType(AMQP_EX_TYPE_DIRECT);
 $exchange-setFlags(AMQP_DURABLE);
 // declare method deprecated. ref http://php.net/manual/fa/amqpexchange.declare.php
 $exchange-declareExchange();
 $this-_exchange = $exchange;
 
 $queue = new \AMQPQueue($this-_channel);
 $queue-setName('xxx.push-notification.que');
 $queue-setFlags(AMQP_DURABLE);
 $args = array('x-ha-policy' = 'all');
 if ($hasDeadLetter) {
     $args['x-dead-letter-exchange'] = 'xxx.err.exg';
     $args['x-dead-letter-routing-key'] = $queueName;
 }
 $queue-setArguments($args);
 $queue-declareQueue();
 $queue-bind('xxx.push-notification.exg', $routingKey);
 
 $params = array('content_type' => 'text/json', 'delivery_mode'=>2);
 $exchange-publish($msg, $routingKey, false, $params);

Golang code

conn, err := amqp.Dial(fmt.Sprintf("amqp://%s:%s@%s:%s/", "guest", "guest", "xx.xx.xx.xx", "5672"))
 if err != nil {
     fmt.Println(err)
     return
 }
 
 ch, err := conn.Channel()
 if err != nil {
     fmt.Println(err)
     return
 }
 
 
 
 out = []byte(`{"command":"sendPushNotification","data":{"userId":"0","profileId":"2701","message":"xxx xxx xxx","section":"general","extraParams":[{"key":"img","val":"aHR0cHM6Ly9zdGWLm00bWFycnkuaW4vdWkvaW1hZ2VzL2xvZ28uZGVza3RvcE5vdGlmaWNhdGlvbi5wbmc="},{"key":"webUrl","val":"https:\/\/lens.xxx.in\/click?evt_id=14d9edf5-7268-463e-8141-03aacf536e65&evt_nm=Clicked+Desktop+Notification&evt_typ=generic&app_id=xxx-jobs&eml_sub=&usr_did=2701&cpg_sc=&cpg_md=&cpg_nm=&cpg_cnt=&cpg_tm=&link_txt=expo+notification+link+clicked&em_type=&cpg_subcode=2021-11-25&cpg_dt=&url=https%3A%2F%2Fstag-fe.xxx.in%2Fsubscription%2FchoosePackage%3Futm_source%3D%26utm_medium%3D%26utm_campaign%3D%26utm_content%3D%26utm_term%3D%26utm_subcode%3D2021-11-25%26evt_typ%3Dgeneric"}],"customFields":[{"key":"pixelTrackerUrl","val":"https:\/\/lens.xxx.in\/i?evt_id=60b675c8-f16d-4224-b3c1-a0b92dccb011&evt_nm=Viewed+Desktop+Notification&evt_typ=generic&app_id=xxx-jobs&eml_sub=&usr_did=2701&cpg_sc=&cpg_md=&cpg_nm=&cpg_cnt=&cpg_tm=&link_txt=&em_type=&cpg_subcode=2021-11-25&cpg_dt="},{"key":"title","val":"xxx team is here in xxx!"}]}}`)
 
 err = ch.Publish("xxx.push-notification.exg", "*", false, false, amqp.Publishing{
     ContentType:  "text/json",
     Body:         out,
     DeliveryMode: amqp.Persistent,
 })

same payload used in both cases.

[mathewvp]

versions

php amqp: 1.10.2

PHP 7.0.33-50+ubuntu20.04.1+deb.sury.org+1 (cli) (built: May 1 2021 11:50:22) ( NTS )

go version go1.13.4 darwin/amd64 (binary is cross compiled for linux)

go amqp github.com/rabbitmq/amqp091-go v1.2.0

Rabbitmq 3.7.14 Erlang 21.3.3

OS Ubuntu 16.04.6

[lukebakken]

For what it's worth, RabbitMQ 3.7.14 has been out of support for a while now. I will be testing using RabbitMQ 3.9.10 and Erlang 24.1.7m

[michaelklishin]

I will convert this issue to a GitHub discussion. Currently GitHub will automatically close and lock the issue even though your question will be transferred and responded to elsewhere. This is to let you know that we do not intend to ignore this but this is how the current GitHub conversion mechanism makes it seem for the users :(

[michaelklishin]

Whether a message is stored in the queue index or message store depends on its size. Message considered small (< 4096 bytes by default) are embedded into the index.

[mathewvp]

The payload when checked from rabbitmq console is 1193 bytes for both cases. It does not display any headers either.

[michaelklishin]

The code above follows two poor practices:

• It hardcodes optional queue arguments in application code, which is extremely inflexible. Use a policy instead
• Classic mirrored queues is a poor practice. "ha-all" is the worst option if you have more than three nodes. Use Quorum queues instead (specifying queue type in application code is perfectly fine)

[mathewvp]

Below is the payload captured using tcpdump

AMQP..........o.
.....1.libraryS....PHP AMQP Lib.library_versionS....2.0.AMQPLAIN...#.LOGINS....guest.PASSWORDS....guest.en_US.........
...................
.(./.............
........'.<.(...xxxxxxx.push-notification.exg.*..........<.............text/json.........{"command":"sendPushNotification","data":{"userId":"0","profileId":"2701","message":"Join+us+at+the+Expo+Center+xxxxxx+from+3+PM+to+9+PM+and+get+amazing+discounts.","section":"general","extraParams":[{"key":"img","val":"aHR0cHM6Ly9zdGFnLWZlLm00bWFycnkuaW4vdWkvaW1hZ2Vzxxxxx28uZGVza3RvcE5vdGlmaWNhdGlvbi5wbmc="},{"key":"webUrl","val":"https:\/\/xxxx.xxxx.in\/click?evt_id=3bfbff26-9af9-4a0c-9a28-17c1e5e1e9d8&evt_nm=Clicked+Desktop+Notification&evt_typ=generic&app_id=xxxxxxx-jobs&eml_sub=&usr_did=2701&cpg_sc=&cpg_md=&cpg_nm=&cpg_cnt=&cpg_tm=&link_txt=xxxxxxx+notification+link+clicked&em_type=&cpg_subcode=2021-12-02&cpg_dt=&url=https%3A%2F%2Fstag-xx.xxxxxxx.in%2Fsubscription%2FchoosePackage%3Futm_source%3D%26utm_medium%3D%26utm_campaign%3D%26utm_content%3D%26utm_term%3D%26utm_subcode%3D2021-12-02%26evt_typ%3Dgeneric"}],"customFields":[{"key":"pixelTrackerUrl","val":"https:\/\/xxxx.xxxx.in\/i?evt_id=128402db-6847-469a-b1c5-01cbdd086126&evt_nm=Viewed+Desktop+Notification&evt_typ=generic&app_id=m4marry-jobs&eml_sub=&usr_did=2701&cpg_sc=&cpg_md=&cpg_nm=&cpg_cnt=&cpg_tm=&link_txt=&em_type=&cpg_subcode=2021-12-02&cpg_dt="},{"key":"title","val":"xxxxxxx team is here in xxxxxxx!"}]}}...........(................
.2........

I still do not understand why this payload takes up 3.8 GB of data for 4300 messages. 1.4 KB x 4300 messages should be around 6 MB

[mathewvp]

This was completely a user side code issue. The messages were being sent in a loop and a bug caused the payload being appended in the loop which led to high payload and subsequent high disk writes. Apologies for raising an invalid issue.

[lukebakken]

I'm glad you figured it out and let us know the cause of the issue. Please remember this next time before opening issues in several different repositories. The vast majority of RabbitMQ and RabbitMQ client library issues are due to bugs in the user's code.