Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

max_bytes_per_partition config mapping within librdkafka #344

Open
lairen opened this issue Jul 26, 2023 · 0 comments
Open

max_bytes_per_partition config mapping within librdkafka #344

lairen opened this issue Jul 26, 2023 · 0 comments

Comments

@lairen
Copy link
Contributor

lairen commented Jul 26, 2023

Should the max_bytes_per_partition config map to librdkafka's fetch.message.max.bytes config instead of message.max.bytes? message.max.bytes is a broker or topic config that seems more relevant to a Kafka producer.

Property C/P Range Default Importance Description
message.max.bytes * 1000 .. 1000000000 1000000 medium Maximum Kafka protocol request message size. Due to differing framing overhead between protocol versions the producer is unable to reliably enforce a strict max message limit at produce time and may exceed the maximum size by one message in protocol ProduceRequests, the broker will enforce the the topic's max.message.bytes limit (see Apache Kafka documentation).
Type: integer
fetch.message.max.bytes C 1 .. 1000000000 1048576 medium Initial maximum number of bytes per topic+partition to request when fetching messages from the broker. If the client encounters a message larger than this value it will gradually try to increase it until the entire message can be fetched.
Type: integer
fetch.max.bytes C 0 .. 2147483135 52428800 medium Maximum amount of data the broker shall return for a Fetch request. Messages are fetched in batches by the consumer and if the first message batch in the first non-empty partition of the Fetch request is larger than this value, then the message batch will still be returned to ensure the consumer can make progress. The maximum message batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (broker topic config). fetch.max.bytes is automatically adjusted upwards to be at least message.max.bytes (consumer config).
Type: integer

This mapping was changed in part because of the error that occurs when setting fetch.max.bytes to a value < message.max.bytes. #163

fetch.max.bytes` must be >= `message.max.bytes` (Rdkafka::Config::ClientCreationError)

This error happens because we are explicitly setting fetch.max.bytes and the value is not being automatically adjusted by librdkafka. After changing the mapping of max_bytes_per_partition -> fetch.message.max.bytes, I was thinking of adding a message_max_bytes config along with validations within Racecar to prevent that error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant