emqx(5.2) source connector

[mntnclmbr]

I have tried implementing the exhook plugin to python client to hstreamdb via the buffered producer setup. While functionally it works, it is not very efficient and ends up loading the emqx servers and slowing down the mqtt pipeline.

Would it be possible to opensource your emqx to hstream connector that is known to have good performance ? I assume it would be written in erlang and be based off the plugin-template. Unfortunately , I am not familiar with erlang and it would take me a long time to get this going on my own.

Appreciate any help I can get with this.

[alissa-tung]

Hello, I suggest to use EMQX enterprise instead of exhook plugins which had done many optimizations for this task. I am not sure what is the hotpoint of the cost of the task, but I guess it may due to the external gRPC call which can not really sent data in batch.
There is no EMQX source connectors for HStream, as the MQTT protocol is not suitable for this settings and can not get good performance at every time.

[alissa-tung]

It would be better if EMQX is equipped with some client functionalities, which means it would be better to write data in batch by EMQX itself, and this is what we do in enterprise version’s.

[mntnclmbr]

is there an hstream erlang client that has the batched producer api ?

[mntnclmbr]

found this : https://github.com/hstreamdb/hstreamdb_erl/blob/main/test/hstreamdb_producer_SUITE.erl

Looks like it will do what I need.

Leaving the reply here in case someone else has a need for it as well.

[alissa-tung]

Sorry for the late reply. Yes, that is the correct API for batch appends.