letsEncrypt listener certificate renewal

[hari819]

I am using 0.27.1 version in the production environment currently , upgrade to 0.31.1 is planned soon .
We are using the ingress type for listeners , it is configured with a certificate from letsencrypt .

We have noticed that the certificate got updated by the cert-manager , but the operator did not reload the brokers which resulted in a severity-1 issue in production environment with all the pods connecting to kafka brokers throwing the authentication error , once i restarted the cluster brokers have taken the updated listener certificate and application is back to working state, pods were able to connect to kafka normally,

I see that the operator will reload the brokers in such scenarios , but any known issue in 0.27.1 version which is the reason for this ,

@scholzj , please could you help , if there is a way we can force the reload of brokers in these cases ,

Thanks,

[hari819]

I saw the discussion here , https://github.com/orgs/strimzi/discussions/7578

but the certificate is created during the creation of the kafka cluster , LetsEncrypt is updating the certificate via the cert-manager , every two months ,

[scholzj]

What exactly does it mean not in a state where the Operator can roll them automatically?

[scholzj]

I mean ... do you have some logs explaining why exactly they cannot be rolled?

[treyhendon]

I've been looking through logs for the past hour and can't find the message. I know I've seen seen something logged previously when this happened. It's similar to "Failed to reconcile", but not that. We're at the halfway point with our certificate lifecycles currently, so I'm a few weeks away from being able to see it happen. I'll post when I can get the log.

[treyhendon]

If it helps, we run as the StrimziPodSet and it's related to the operator not thinking it can do a rolling restart.

[scholzj]

Without the exact log, it is really hard to say what could have gone wrong. So I'm afraid there is not much more I can do until it happens next time and you can get the exact log. One option is that the replicas are not in sync and we cannot roll the pods while maintaining availability. But not sure that would be your case - there are of course other possibilities as well.