The goal is to clean up unused topics in Kafka
Unused topics are topics that match the following three conditions:
- Not
internal
that used by kafka components. - Topics have not been consumed for a while, that the consumers are already cleaned up by kafka, which by default is 7 days.
- Topic that is empty
BOOTSTRAP_SERVERS
: broker hostsIS_DELETE_ENABLED
: without setting this, this application does not delete any topic and only log the topics should be deleted
docker run --rm --name kafka-topic-cleaner --network=host -e BOOTSTRAP_SERVERS=localhost:29092 sendohchange/kafka-topic-cleaner
mvn clean package
docker build -t kafka-topic-cleaner .
docker run --rm --name --network=host kafka-topic-cleaner -e BOOTSTRAP_SERVERS=localhost:29092 kafka-topic-cleaner
ordocker run --rm --name --network=host kafka-topic-cleaner -e BOOTSTRAP_SERVERS=localhost:29092 -e IS_DELETE_ENABLED=true kafka-topic-cleaner
Use docker-compose
and it creates some sample topics that demonstrates deletions
To know which topics are/were consumed, the library must implement ListConsumerGroup API. Only one python library is up-to-date, developed by confluent, but this API does not implement ListConsumerGroup API yet. confluentinc/confluent-kafka-python#223