Zeebe Kafka Exporter 2.0.0
2.0.0
This new Zeebe Kafka Exporter releases is the first Zeebe 0.23+ compatible release. It was manually tested with both 0.23.x (up to 0.23.4) and 0.24.x (up to 0.24.1). It is not compatible with Zeebe versions lower than 0.23.
Changes
New serialization format
I've dropped the Protobuf format and replaced it with JSON. The previous format required users to use the specific Protobuf schema but wasn't well integrated with the Kafka ecosystem (e.g. via Schema Registry), was not human readable, was hard to use with Kafka Connect out of the box, and generally made for a worse user experience.
While JSON is schemaless, it addresses the points above by being easy to read, easy to use in the Kafka ecosystem, and doesn't require any additional dependencies to serialize/deserialize. The serde module included is entirely optional and can be ignored, and users will still be able to make use of the published records.
Configuration changes
On top of updating the configuration for Zeebe 0.23+, there were previously some issues with the free-form producer configuration. The format was changed now to avoid any issues by delegating the processing to Kafka. Essentially, to overwrite or add additional producer configuration (e.g. to support SASL, or OAuth), you specify the configuration as a properties file. So in your YAML file, you would add it as a multi-line string which would be the contents of your normal producer properties file. The README contains an example for this.
Minor improvements
A few other touchups were done as part of the 2.0 release. The code is now fully documented, and a new QA module was added, replacing the old examples module. The QA module now tests the exporter in an as-close-to-production deployment as possible, by leveraging Testcontainers to create a new Zeebe container, a new Kafka container, and injecting the exporter JAR into the Zeebe container along with its configuration file. The tests included are also a good way to show how you can configure the exporter through a mix of YAML and environment variables. It also acts as a proof of concept for exporter authors on how to test their own exporters.