Strimzi Kafka HPA integration

[pashafirdous1]

Hi Team,

we are interested to know if there is any support for Strimzi Kafka to support integrating with HPA(Horizontal pod autoscaler)?

we want to scale Kafka broker pods based on thresholds limits reached. for ex. consumer lag latency, CPU/Memory thresholds

there are many implementations with kafka out of operator framework.
but please let us know if we have any method to scale strimzi kafka with operator.

regards

[scholzj]

No, we do not support it for Kafka. Only for other resources such as Connect etc.

Scaling Kafka based on the metrics mentioned above is not so easy. Just adding more nodes does not solve anything. If nothing else, you would need to redistribute the partition replicas to the new brokers. And even that might not necessarily solve for example too big consumer lag. Similarly for scaling down - you would need to move the partition replicas first before scaling down. The redistribution of partitions would take some time (possibly quite a lot of time for big brokers), so you would need to make sure you scale it for the long-term and not just for 5 minutes. So it is fairly complicated.

One day we might be able to do the partition redistribution automatically which would make it easier, but that is not supported yet.

[pashafirdous1]

can we scale pods and use cruise control to manage the partition redistribution?

[scholzj]

Yeah, sure, you can do that. Just use the KafkaRebalance resource after scaling up.