Is changing storage class supported for strimzi after a strimzi kafka cluster is built?

[dbaarorapr]

We recently lost some namespace and it had a strimzi kafka cluster running in it

Now ,we were using deleteClaim: false

As we lost namespace ,we also lost our PVC's and since the storage class we were using had reclaimPolicy: Delete which meant that we lost all PV also

Now when we create a new storage class which has reclaimPolicy: Retain and use it in the kakfa CR objects in strimzi ,we see it complains with in the kubectl describe kafka

  Conditions:
    Last Transition Time:  2022-06-10T15:43:14.051971Z
    Message:               The desired ZooKeeper storage configuration contains changes which are not allowed. As a result, all storage changes will be ignored. Use DEBUG level logging for more information about the detected changes.
    Reason:                ZooKeeperStorage
    Status:                True
    Type:                  Warning
    Last Transition Time:  2022-06-10T15:43:14.021209Z
    Message:               The desired Kafka storage configuration contains changes which are not allowed. As a result, all storage changes will be ignored. Use DEBUG level logging for more information about the detected changes.
    Reason:                KafkaStorage
    Status:                True
    Type:                  Warning
    Last Transition Time:  2022-06-10T15:43:17.078Z
    Status:                True
    Type:                  Ready

strimzi logs shows below

2022-06-10 16:01:46 WARN  AbstractModel:439 - Reconciliation #13091(timer) Kafka(kafka/test-kafka): Only the following changes to Kafka storage are allowed: changing the deleteClaim flag, adding volumes to Jbod storage or removing volumes from Jbod storage, changing overrides to nodes which do not exist yetand increasing size of persistent claim volumes (depending on the volume type and used storage class).

I plan to always patch the underlying PV's with a cronjob on the hour and keep using the same old storage class as a workaround

kubectl patch pv $vol_name -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'; 

How do others handle this ?

[scholzj]

No, it is not supported because Kubernetes does not support it. And in general it cannot be supported because changing storage class might mean completely changing the characteristics of the volume. You cannot change SSD into spinning drive or block storage into NFS just by changing some field.

To fix the Strimzi warning, you can fix the annotation on the SattefulSets / StrimziPodSets. But that does not solve the underlying PVCs / PVs.

[dbaarorapr]

Thanks for clarifying

[kinihun]

Since Strimzi is an operator, it can orchestrate a process of deleting brokers, and associated volumes and recreating them, waiting for the topics to be well replicated and then proceed to the next broker in in a rolling update. What do you think about my wild idea :)

[kinihun]

My clusters were originally created using Standard_LRS-backed volumes, overtime, my brokers have become unevenly distributed across availability zones. AKS still tries to distribute the nodes across all the zones but my broker pods aren't getting schedule based on my topology spread constraint and node affinity because the volumes are fixed in location. It would be great to be able to change to ZRS without having to teardown the cluster.

[scholzj]

In theory yes, the operator could do it the way you suggest. But it does not and I think it might be a bad practice. Doing it this way can take a very long time. During that time, the operatormight be basically paralysed waiting for the topics to resync. It does not also make the consequences clear to the user => that during such operation, the availability and reliability will be weaker because of that.

So, if you really know what you are doing and what the consequences are, you can do this your self semi-manually. But the operator will not expose this to any users running into this at random without the understanding.