BE: Expose consumer's config props likerequest.timeout.ms
#527
Comments
cpapad
added
status/triage
kapybro
bot
added
status/triage/manual
|
Hi cpapad! 👋 Welcome, and thank you for opening your first issue in the repo! Please wait for triaging by our maintainers. As development is carried out in our spare time, you can support us by sponsoring our activities or even funding the development of specific issues. If you plan to raise a PR for this issue, please take a look at our contributing guide. |
Haarolean
added
type/enhancement
Haarolean
changed the title
Haarolean
changed the title
request.timeout.ms
|
Facing a similar problem need the ability to configure |
|
I'm facing exactly the same problem as @Jaideep-C . |
|
@Haarolean, I faced the same issue, happy to contribute if it's still up for grabs. |
|
Hi @Alex1OPS How are you approaching to expose the configs? |
|
@Jaideep-C According to ConsumerGroupService you can specify the necessary consumer settings via the cluster properties, for example: Can you test this? |
|
@wernerdv This would work, but cluster properties are also applied to the admin client, which can be undesirable. For example, the Another pain point is that in the current implementation whatever is specified in cluster property kafka:
adminClientTimeout: 75000
clusters:
- name: local
bootstrapServers: localhost:9092
properties:
isolation.level: read_committed
request.timeout.ms: 600000because admin client timeout will be set to I see three options:
|
|
I've took a look at the code and think that the isolation.level=read_commited won´t solve the message count presented, since it appears that it is calculated by the sum of last offsets of each topic / partition. Above all, this is done by the admin client and not a consumer and, as far as I know, admin client does not support / use the property. |
No matter if exposing consumer configs gonna solve this issue, we still plan on doing so (there's another case #793 calling for this). |
Let's proceed with the third option. Could you also allow overriding both producer and consumer properties in your PR to address #793 as well? |
Issue submitter TODO list
Is your proposal related to a problem?
When we query with CEL filters or free text large topics we come up with the following error
[Consumer clientId=kafbat-ui-consumer-1724336921133, groupId=null] Failed to close fetcher with a timeout(ms)=30000To overcome this we need to override the consumer request time out and other values
While admin configuration options are exposed as far as we are concerned we can not override with configuration or environment variables timeout settings e.t.c. of the consumer.
Describe the feature you're interested in
Values to override request timeout and relevant settings of the consumer used for querying topics to avoid errors and timeouts in large topics.
Describe alternatives you've considered
No response
Version you're running
2956664 v1.0.0
Additional context
No response
The text was updated successfully, but these errors were encountered: