Help on Disk Usage - Few PVCs are getting full and getting "No Space Left on Device"

[npandeya]

Hello,
I am running the cluster with 30 brokers. Each broker has this storage spec:

storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        class: kafka
        size: 750Gi
        deleteClaim: false

log retention hours is 12 hours. with this config log.retention.hours

What I am seeing is only few of the pods is filling up fast. Is there a way I can balance the disk storage across all the PVCs? I was reading up on kafkarebalance but only saw the specs etc. Will using cruisecontrol with the kafkarebalance will help? I was thinking of using the default configuration for both.

Thank you for your help.

[scholzj]

You can use KafkaRebalance. You just need to specify the storage goal in its spec (IIRC they start with Disk in the name?). Please check the Strimzi and Cruise Control docs. Keep in mind that a single partition is always assigned to one broker. So if you have some huge partitions or something like that, Cruise Control might not be able to help.

You can also configure bytes retention for your topics or increase the disk size.