Add options for librdkafka partitioning compatibility #455
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
owenniles
commented
Jul 15, 2024
There was a problem hiding this comment.
The changes in this file were inspired by the implementation of this function.
owenniles
force-pushed
the
owen/librdkafka-compat
branch
from
3b27a4d to
0f5c477
Compare
|
It occurred to me that setting the partitioning behavior for each lookup table may be desirable as well #456. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR allows developers to configure Goka's partitioning behavior to match librdkafka. Open to feedback on the approach!
Why?
To allow the group table to be copartitioned with input topics whose producers use librdkafka's partitioning behavior. There is a subtle difference in the way librdkafka and Sarama convert hashes of keys to partition numbers (embarrassingly, here is my source for this claim). If a Goka processor processing an input topic with librdkafka-based producers uses sarama's partitioning behavior, the group table will not be copartitioned with the input topics.
Changes
goka.ProducerBuilderconstructor that allows developers more granular control over the producer'ssarama.HashPartitioner, which is responsible for assigning keys to partitions.sarama.HashPartitionerby writing a customgoka.ProducerBuilder, but I believe it would be nicer if Goka provided a first-party way to do this.goka.WithViewHashUnsignedthat configures the view to calculate the number of the partition in which a key can be found in the same way librdkafka would.