Consumer Metrics Not Showing

[rajpootneelesh]

Issue: Metrics Not Available When Consumers Are Stopped in Kafka Lag Exporter

Environment:

• Kafka Version: 3.6.1
• Kafka Lag Exporter Version: 1.8.0 (2024-08-20)

Description:
I am currently using Kafka Lag Exporter to monitor consumer lag. However, I have noticed that metrics for a consumer group are not available when the consumer is stopped or disconnected. The metrics only appear once the consumer is actively running again.

Previously, with Kafka Version 2.7.0, this issue did not occur. Metrics for the consumer group were always available regardless of the consumer's state (running or stopped/disconnected).

This change in behaviour is affecting our ability to monitor idle or disconnected consumers effectively.

Steps to Reproduce:

1. Deploy Kafka Lag Exporter with Kafka 3.6.1.
2. Start a consumer group and verify that metrics are visible in Kafka Lag Exporter.
3. Stop the consumer group.
4. Check metrics for the stopped consumer group in the exporter.

Expected Behaviour:
Metrics for the stopped consumer group should still be visible in Kafka Lag Exporter.

Actual Behaviour:
Metrics disappear as soon as the consumer group stops and only reappear when the consumer is running again.

Additional Notes:
This issue was not present in Kafka 2.7.0 with the same Kafka Lag Exporter setup.

Request:
Could you help identify if this is a known issue or a compatibility problem with Kafka 3.6.1? Are there any configuration changes or workarounds that can resolve this behaviour?