Operator not creating ownerReferences on PersistentVolumeClaims

[Tommyf]

Hi,

I have provisioned a persistent Kafka cluster, based on the example here:

The pertinent section on storage is identical:

storage:
  type: jbod
  volumes:
  - id: 0
    type: persistent-claim
    size: 10Gi
    deleteClaim: false

When the Kafka cluster is created, the PersistentVolumeClaim manifest generated by the controller does not contain anownerReferences section.

While not a problem in and of itself, it does cause a problem with some toolings (e.g. ArgoCD) that look at resources in the cluster and use the ownerReferences to build a dependency graph. As the PVCs have no listed owners, they look like dangling resources that should be pruned.

It's possible to get ArgoCD to ignore these resources, but that does not seem like the correct approach to me.

Is there a reason that the PersistentVolumeClaims created by Strimzi do not have ownerReferences added?

I guess there are three possible answers to this.

1. No, Strimzi should not be adding ownerReferences to the PestistentVolumeClaim. If this is the case, I would like to understand the reasoning for this.
2. No, not by default, but there is an option to enable it. I could not find such an option so I would love for someone to point me in the right direction.
3. This is a bug, it should be adding it. In this case, I will open an issue for this.

I wanted to ask first before opening an issue in case the answer is 1 or 2.

[scholzj]

The right answer is mix of 1 and 2.

Kubernetes by default do not delete the PVCs. This is useful for example to not lose all your data when someone by mistake deletes the Kafka custom resource. When you set the owner reference, the PVCs and possibly with it also the PV (depends on other configurations as well) will be deleted in such case. So yes, by default we do not set the owner reference and this is fully intentional and by design. If you check StatefulSets, they also don't use owner references in the PVCs they create. So we are completely consistent in this behavior and follow the Kubernetes defaults.

If you really want to have the the ownerReference there and have the PVC deleted when the Kafka CR is deleted, you can use the deleteClaim flag to true (you have it in the YAML snippet above). This will set the ownerReference and this will delete the PVCs together with the Kafka cluster. I normally recommend this only for development to safe costs due to left-over PVCs and not for production. But at the end it is your choice. But I think you should consider the consequences of it carefully and not just do it to solve any issues with Argo.

[Tommyf]

Interesting about the StatefulSets not setting ownerReferences. I will need to look more into how ownerReferences are used in general so thanks for that pointer.

I did not think that the ownerReferences were specifically to orchestrate deletion of owned resources when an owner resource is deleted. It seemed to me more like building a resource hierarchy (certainly seems to be used this way in the ArgoCD case).

I was comparing the behaviour with that of the CrunchyData PGO. With that operator, the PVC's do have ownerReferences set. They have PVC deletion protected by including the blockOwnerDeletion: true stanza as well as the kubernetes.io/pvc-protection finalizer.

I really don't want to set the deleteClaim flag to true, so is there no other way to set the ownerReferences, but with blockOwnerDeletion set to true to prevent the deletion problem?

Clearly I do need to brush up on ownerReferences and Finailizers and such as I just re-read the docs and it seems that maybe ownerReferences IS about resource deletion :)

I also just found this, that I did not find before: argoproj/argo-cd#10096
Relevant, as this is the problem I was encountering, so I will investigate that approach also.

[scholzj]

I think the Kubernetes Garbage Collection which is the deletion of the resources when the owner is deleted was here before Argo. So I think it is more likely that it was the original concept behind it. But TBH, it does not matter because what matters is how it works and what effect it has. 🤷

The blockOwnerDeletion: true does not have the same effect. It does not block the deletion of the PVC when the owner is deleted. It blocks the deletion of the owner. So this is completely different. We do not want the PVCs to block intentional deletions of the Kafka CRs.

Finalizers have its own share of troubles in the way how they can break different things. But in general, if you want you can use them -> you can for example enable the deleteClaim flag to get the owner reference in the PVCs but block the deletion of the the Kafka CR with your own custom finalizer to avoid cluster deletion by mistake.

From my point of view, the problem is that Argo takes a lot of assumptions and tries to force them on everyone else. Take for example the issue you linked. It suggests some labels such as app.kubernetes.io/instance should not be copied to the PVCs. But then there are many other applications - for example for cost management, inventory management or security purposes - which want to track the resources belonging to an application instance including the PVCs (which are important source of costs or might store sensitive data and thus be very relevant for security monitoring).

I would be more then happy to improve Strimzi to work better with Argo. But it cannot be at the cost at breaking things for other users. It would be for example great if there was some annotation such as argo-cd.io/ownerReference which could be used as an alternative to the Kubernetes owner references for cases when you don't want to set the real owner reference. That would not interfere with any other projects and I would be more than happy to have it set by default on the PVCs. But as far as I know (with my very limited Argo knowledge) there is nothing like that what we can use.

In some other discussions here on the Strimzi GitHub, I think some users suggested that some annotations might be used to tell Argo to ignore the PVCs and leave them be. Unfortunately, that did not seem to be something everyone wants set by default either. But at least you can set it through the template section if this is what you want (e.g. in .kafka.spec.template. persistentVolumeClaim).