The API design of AvroEncoder

[mirosval]

Hello and thank you for all the hard work on this library!

I'd like to use it in an async project. Its a web server that accepts some payload and then forwards it to Kafka using Avro. In my code I have a KafkaClient that accepts a struct and is supposed to send it to kafka. The client and the web server have the same lifetime; the client gets cloned for every request the web server receives.

The problem I'm running into is that AvroEncoder::encode_struct takes a mutable reference to self, which means it can't be cloned. The Schemas are cached internally and there seems to be no way of providing a schema upfront (like the result of async_impl::schema_registry::post_schema) but it gets resolved every time on cache miss. Because of that there is seemingly no way how I can efficiently use AvroEncoder. The only way to make it work is to create it new on every message send, which will make a call to the schema registry. Instead I would like to initialize it once with a schema I supply and then use it.

Am I missing something?

[gklijs]

I still have not build a full async app to test the usability in that case so it could be there is room for improvement.

However you should not create a new encoder for each request, as it will cause the need for a call to schema registry each time. The problem is you need the correct ID from schema registry, even if you supply a schema.

From a theoratic view there seem at least two options:

• Wrap the encoder in a mutex, this way you can use the same one from multiple threads, but might slow things down.
• Use a separate encoder for each thread. You will have cache misses, and additional calls to schema registry. But the thread never needs to wait till another thread is done. With the cache filled, the tread should never have to wait long, but for extreme performance this might be needed.

I hope this helps somewhat. I might add some example app some point, I think it might be useful.

[mirosval]

Thank you very much. Actually, you're probably right, wrapping it in a mutex is the way to go for me, because 99% of the time it will be a cache hit, so the slowdown should be minimal. Still if I may, I would suggest decoupling the cache from the encoder, as it would make it a lot more usable. Thanks again!

[gklijs]

I will think about it, the cache is one of the most usefull/complex bits. It's also the only place that needs mutability. So decoupling might make some things easier in the end. For context, the library has grown from just a passion project trying to do the same thing with Rust, as I was doing with Rust. I'm not using it myself at the moment, and try to keep it compatible with the Java client if needed, but don't feel like doing much more for it. If you want to work on it, you could create an issue and pr for it.

[mirosval]

I just looked through the code and I think it would be sufficient to make the following items public so I can directly access them:

• async_impl::avro::to_avro_schema
• avro_common::item_to_bytes
• avro_common::AvroSchema

Would you accept a PR just doing that? Then I can call post_schema to get the RegisteredSchema, then to_avro_schema to get at AvroSchema which I can use directly in item_to_bytes to serialize my struct.

[gklijs]

Have to think about it for a bit. Would be a bit weird to do just that, as I like to keep the different flavours as much the same as possible. Also exposing more in the api means more risk of breaking the api eventually.

[mirosval]

Actually I found a way:

        let schema: RegisteredSchema = ???
        let value = avro_rs::to_value(message)?;
        let inner = avro_rs::to_avro_datum(&schema.schema, value)?;
        let payload =
            schema_registry_converter::schema_registry_common::get_payload(schema.id, inner);

Thank you for you help.