Organize operators to perform deletion operations in abnormal scenarios

[lanzhiwang]

Strimzi kafka operator 0.18.0 was deployed in our cluster. By checking the logs, we found that some topics were accidentally deleted. The specific logs are as follows:

[2023-08-14 10:16:25,803] INFO <opicOperator:535> [oop-thread-0] 1761120|periodic kafka xxx|241001519: Reconciling topic xxx, k8sTopic:nonnull, kafkaTopic:null, privateTopic:nonnull

The above log shows that when the operator requests topic information from kafka, kafka does not correctly return the topic that exists in kafka. At this time, there is a corresponding kafkatopic cr in k8s, which causes the operator to perform subsequent deletion operations.

If you only go here, there is no problem, because at this time the operator will only delete /strimzi/topic in zk and kafkatopic cr, and will not actually delete the topic under /broker/topic

However, there is still an accident at this time. Deleting kafkatopic cr by the operator will trigger another reconcile. At this time, the state seen by the operator may be

k8sTopic:null, kafkaTopic:nonnull, privateTopic:nonnull

This is to actually delete the topic under /broker/topic

The above is an inference based on the implementation logic of the operator. I tried to reproduce this problem in another environment. After confirming that the topic existed in kafka, I found that the operator could not obtain the topic from kafka because at this time I forced The kafka pod was deleted. Although the topic under /broker/topic was not deleted in the end, and the kafkatopic cr was automatically rebuilt in the end, I understand that the above inference is possible.

According to the current implementation logic, the topic obtained by the operator is empty, and it is not sure whether it is caused by an exception or the topic is really deleted

In order to prevent the topic from being accidentally deleted, I suggest adding a field in the topic operator that indicates whether to do the final deletion operation, such as deleteTopicEnable=true

   entityOperator:
     topicOperator:
       deleteTopicEnable: false
       jvmOptions:
         -Xms: 500m
         -Xmx: 500m
       record:
         loggers:
           rootLogger.level: TRACE
         type: inline
       reconciliationIntervalSeconds: 15
       resources:
         limits:
           cpu: "1"
           memory: 2Gi
         requests:
           cpu: "1"
           memory: 2Gi

Then delete it according to the existing logic. If deleteTopicEnable=false, then the operator will not delete kafkatopic cr, topic under /broker/topic, topic under /strimzi/topic under any circumstances.

Use this field to allow users to confirm what to do for deletion, and allow users to assess the risk of deletion themselves.

[scholzj]

You should check how the Unidirectional Topic Operator works. That is what Strimzi plans to use as the Topic Operator in the future. We do not plan any further changes to the Bidirectional Topic Operator.

[lanzhiwang]

You mean that the operator will not synchronize the topic in kafka to k8s in the future. when will this happen

[scholzj]

This is available as a feature gate.

[lanzhiwang]

Has this function been implemented? It seems that I haven’t seen the documentation. Is there any documentation for this?

[scholzj]

Yes, as I said, it is available as a feature gate. Check the feature gates documentation and look for the UnidirectionTopicOperator feature gate. It is of course not in Strimzi 0.18 which is several years old.

[lanzhiwang]

thanks