forked from wushujames/kafka-utilities
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add docs. Move who_in_isr docs around.
- Loading branch information
1 parent
cda8cce
commit 82adb5a
Showing
3 changed files
with
113 additions
and
47 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| # ConsumerGroupLag | ||
| A faster alternative to `kafka-consumer-groups.sh --describe`, that can also output the information as JSON. Useful for piping into scripts. | ||
|
|
||
| Timing on a consumer group that consumes 800 partitions from a remote Kafka cluster: | ||
| kafka-consumer-groups.sh 128 seconds | ||
| ConsumerGroupLag 5 seconds | ||
|
|
||
| By default, it outputs in a format that is (mostly) compatible with kafka-consumer-groups.sh from Kafka 0.10.1.1 | ||
|
|
||
| ``` | ||
| GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER | ||
| wushujames-1491549464 some-topic 0 unknown 531220534 unknown consumer-1_/192.168.1.1 | ||
| wushujames-1491549464 some-topic 1 523573472 523573472 0 consumer-1_/192.168.1.1 | ||
| wushujames-1491549464 some-topic 2 521553458 521553468 10 consumer-1_/192.168.1.1 | ||
| ``` | ||
|
|
||
| If you add `--include-start-offset`, it adds an additional column that shows the first offset in each partition | ||
| ``` | ||
| GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER LOG-START-OFFSET | ||
| wushujames-1491549464 some-topic 0 unknown 531220534 unknown consumer-1_/192.168.1.1 0 | ||
| wushujames-1491549464 some-topic 1 523573472 523573472 0 consumer-1_/192.168.1.1 0 | ||
| wushujames-1491549464 some-topic 2 521553458 521553468 10 consumer-1_/192.168.1.1 0 | ||
| ``` | ||
|
|
||
| If you add `--json` or `-J`, it will output the information as JSON. Useful for piping into scripts. | ||
| ``` | ||
| { | ||
| "some-topic" : { | ||
| "0" : { | ||
| "logStartOffset" : 0, | ||
| "logEndOffset" : 531145685, | ||
| "partition" : 0, | ||
| "currentOffset" : "unknown", | ||
| "lag" : "unknown", | ||
| "consumerId" : "consumer-1-f730ab1e-cdf0-41c4-b2c2-dd195ad40b4d", | ||
| "host" : "/192.168.1.1", | ||
| "clientId" : "consumer-1" | ||
| }, | ||
| "1" : { | ||
| "logStartOffset" : 0, | ||
| "logEndOffset" : 523505602, | ||
| "partition" : 1, | ||
| "currentOffset" : 523505602, | ||
| "lag" : 0, | ||
| "consumerId" : "consumer-1-f730ab1e-cdf0-41c4-b2c2-dd195ad40b4d", | ||
| "host" : "/192.168.1.1", | ||
| "clientId" : "consumer-1" | ||
| }, | ||
| "2" : { | ||
| "logStartOffset" : 0, | ||
| "logEndOffset" : 521484818, | ||
| "partition" : 2, | ||
| "currentOffset" : 521484808, | ||
| "lag" : 10, | ||
| "consumerId" : "consumer-1-f730ab1e-cdf0-41c4-b2c2-dd195ad40b4d", | ||
| "host" : "/192.168.1.1", | ||
| "clientId" : "consumer-1" | ||
| } | ||
| } | ||
| } | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| # who_in_isr | ||
| When I have under replicated partitions in my kafka cluster, this handy utility lets me know if a particular broker is at fault. | ||
|
|
||
| Requires kafkacat, and in particular, a version that supports json output (-J) | ||
| ``` | ||
| $ kafkacat -L -b broker.example.com -J | who_in_isr.py | ||
| Broker 1 | ||
| 1 in isr: 100.00% | ||
| 2 in isr: 100.00% | ||
| 3 in isr: 100.00% | ||
| 4 in isr: 0.00% | ||
| 5 in isr: 100.00% | ||
| Broker 2 | ||
| 1 in isr: 100.00% | ||
| 2 in isr: 100.00% | ||
| 3 in isr: 100.00% | ||
| 4 in isr: 10.87% | ||
| 5 in isr: 100.00% | ||
| Broker 3 | ||
| 1 in isr: 100.00% | ||
| 2 in isr: 100.00% | ||
| 3 in isr: 100.00% | ||
| 4 in isr: 0.00% | ||
| 5 in isr: 100.00% | ||
| Broker 4 | ||
| 4 in isr: 100.00% | ||
| Broker 5 | ||
| 1 in isr: 100.00% | ||
| 2 in isr: 100.00% | ||
| 3 in isr: 100.00% | ||
| 4 in isr: 0.00% | ||
| 5 in isr: 100.00% | ||
| Broker -1 | ||
| 1 in isr: 0.00% | ||
| 4 in isr: 0.00% | ||
| 5 in isr: 0.00% | ||
| ``` | ||
| How to read this: | ||
| * For partitions with leaders on broker 1, broker 4 is missing from the ISR for all the replicas it is supposed to be part of. | ||
| * For partitions with leaders on broker 2, broker 4 is only in the ISR for 10.87% of the replicas it is supposed to be part of. | ||
|
|
||
| Looking at all brokers, it is always broker 4 that is missing from ISRs. This indicates that it is likely that broker 4 is the one having problems. | ||
|
|
||
| Broker -1 means that there are some partitions have no leaders. They are likely offline. |