Strimzi is not honoring the reason for restart events during the certificate renewal

[saeedya]

Hi
When the new cert renewal is happening , we are expecting the events as below:
CaCertRenewed CaCertRenewed CA certificates have been renewed, and the pod is restarted to run with the updated certificates.
Restart reasons
But we are getting "Pod has old revision".
@scholzj
Thanks
Saeed

[scholzj]

I'm not sure this is unexpected. To make sure that the rolling update happens even when the in-memory state of the operator is lost, the certificate versions are tracked through annotations and when they change, that creates the Pod has old revision restart reason. So this does not seem necessarily wrong - at leats not without seeing the exact logs and understanding the actual scenario.

[scholzj]

Yes, for example.

[saeedya]

I have all those annotation, but when the client ca cert renewal happens still the annotation "strimzi.io/clients-ca-cert-generation: 0" and won't change. Pods are getting restarted but in the events i am getting "PodHasOldRevision" message not "CaCertRenewed".

[scholzj]

Again, you would need to explain the exact scenarios, configs, and logs. You are asking for a very detailed explanations to a very generic questions.

[saeedya]

apiVersion: v1
items:
- apiVersion: kafka.strimzi.io/v1beta2
  kind: Kafka
  metadata:
    annotations:
      jmx-filter-configmap-name: eric-data-message-bus-kf-metrics-config
      mydomain.io/app: msg1
      mydomain.io/instance: msg1-instance
    creationTimestamp: "2024-10-18T15:14:53Z"
    generation: 16
    labels:
      application: msg1
      eric-data-distributed-coordinator-ed-access: "true"
      eric-sec-key-management-access: "true"
      eric-sec-sip-tls-access: "true"
      instance: msg1-instance
    name: eric-data-message-bus-kf-test
    namespace: saeed-op
    ownerReferences:
    - apiVersion: mbkf.data.ericsson.com/v1
      blockOwnerDeletion: false
      controller: false
      kind: MessageBus
      name: eric-data-message-bus-kf-test
      uid: bf8ba719-16ce-4704-8d13-81bfd477fd17
    resourceVersion: "2115903355"
    uid: 9899c51d-b8ec-401f-82d1-d7a202bf4526
  spec:
    entityOperator:
      template:
        pod:
          affinity:
            podAntiAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
              - podAffinityTerm:
                  labelSelector:
                    matchLabels:
                      strimzi.io/kind: Kafka
                      strimzi.io/name: eric-data-message-bus-kf-test-kafka
                  topologyKey: kubernetes.io/hostname
                weight: 100
          imagePullSecrets:
          - name: armdocker
          metadata:
            annotations:
              mydomain.io/app: msg1
              mydomain.io/instance: msg1-instance
            labels:
              application: msg1
              eric-data-distributed-coordinator-ed-access: "true"
              eric-sec-key-management-access: "true"
              eric-sec-sip-tls-access: "true"
              instance: msg1-instance
          securityContext:
            fsGroup: 10000
            runAsNonRoot: true
          tolerations:
          - effect: NoSchedule
            key: key1
            operator: Equal
            value: value1
          topologySpreadConstraints:
          - labelSelector:
              matchLabels:
                strimzi.io/kind: Kafka
                strimzi.io/name: eric-data-message-bus-kf-test-entity-operator
            maxSkew: 1
            topologyKey: zone
            whenUnsatisfiable: ScheduleAnyway
        userOperatorContainer:
          env:
          - name: METRICS_JMX_ENABLED
            value: "true"
          - name: METRICS_JMX_PORT
            value: "9404"
          - name: LOG4J_LOG_LEVEL
            value: INFO
          - name: STRIMZI_CUSTOM_ENTITY_OPERATOR_LABELS
            value: 'app.kubernetes.io/version: 3.5.0-ha4ca65e3'
          - name: HELMCHART_VERSION
            value: 3.5.0-ha4ca65e3
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            privileged: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true
      userOperator:
        additionalVolumes:
        - mountPath: /opt/kafka/certificates/eric-sec-sip-tls-trusted-root-cert
          name: eric-sec-sip-tls-trusted-root-cert
          targetContainers:
          - user-operator
          volumeSource:
            secret:
              items:
              - key: ca.crt
                path: ca.crt
              secretName: eric-sec-sip-tls-trusted-root-cert
            type: secret
        logging:
          loggers:
            appender.console.layout.pattern: '{"version": "1.2.0", "timestamp": "%d{yyyy-MM-dd''T''HH:mm:ss.SSSZZZZZ}",
              "severity": "%p{WARN=warning, DEBUG=debug, ERROR=error, FATAL=critical,
              TRACE=debug, INFO=info}", "service_id": "eric-data-messagebus-kf", "message":
              "%enc{%m}{JSON}%enc{%ex{separator(|)}}{JSON}", "metadata": {"container_name":
              "user-operator", "pod_name": "$${env:HOSTNAME}"}, "extra_data": {"file":
              "%file", "line": "%line", "mbkf_cluster_name": "eric-data-message-bus-kf-test"}}%n'
            appender.console.layout.type: PatternLayout
            appender.console.name: consoleLogger
            appender.console.target: SYSTEM_OUT
            appender.console.type: Console
            rootLogger.additivity: "false"
            rootLogger.appenderRef.console.ref: consoleLogger
          type: inline
        resources:
          limits:
            cpu: "2"
            memory: 1Gi
          requests:
            cpu: "1"
            memory: 512Mi
    kafka:
      additionalVolumes:
      - mountPath: /opt/kafka/certificates/tls-siptls-9093-certs/eric-data-message-bus-kf-test-client-ca-secret
        name: siptls-9093-eric-data-message-bus-kf-test-client-ca-secret-ca-certs
        targetContainers:
        - kafka
        volumeSource:
          secret:
            items:
            - key: clientcacertbundle.pem
              path: ca.crt
            secretName: eric-data-message-bus-kf-test-client-ca-secret
          type: secret
      - mountPath: /opt/kafka/certificates/eric-sec-sip-tls-trusted-root-cert
        name: eric-sec-sip-tls-trusted-root-cert
        targetContainers:
        - kafka
        volumeSource:
          secret:
            items:
            - key: ca.crt
              path: ca.crt
            secretName: eric-sec-sip-tls-trusted-root-cert
          type: secret
      - mountPath: /opt/kafka/certificates/metrics-server-secret
        name: metrics-server-cert
        targetContainers:
        - kafka
        - zookeeper
        volumeSource:
          secret:
            items:
            - key: clicert.pem
              path: clicert.pem
            - key: cliprivkey.pem
              path: cliprivkey.pem
            secretName: eric-data-message-bus-kf-metrics-server-secret
          type: secret
      - mountPath: /opt/kafka/certificates/eric-pm-server-ca
        name: eric-pm-server-ca
        targetContainers:
        - kafka
        - zookeeper
        volumeSource:
          secret:
            items:
            - key: client-cacertbundle.pem
              path: ca.crt
            secretName: eric-pm-server-ca
          type: secret
      - mountPath: /opt/kafka/certificates/tls-externaltls-9094-certs/eric-bss-msg-kafka-external-kafka-client-ca-0
        name: externaltls-9094-eric-bss-msg-kafka-external-kafka-client-ca-0-ca-certs
        targetContainers:
        - kafka
        volumeSource:
          secret:
            items:
            - key: ca-bundle.crt
              path: ca-bundle.crt
            secretName: eric-bss-msg-kafka-external-kafka-client-ca-0
          type: secret
      - mountPath: /opt/kafka/certificates/tls-externaltls-9094-certs/eric-bss-msg-kafka-external-kafka-client-ca
        name: externaltls-9094-eric-bss-msg-kafka-external-kafka-client-ca-ca-certs
        targetContainers:
        - kafka
        volumeSource:
          secret:
            items:
            - key: ca-bundle.crt
              path: ca-bundle.crt
            secretName: eric-bss-msg-kafka-external-kafka-client-ca
          type: secret
      authorization:
        superUsers:
        - CN=eric-adp-mbkf-superuser
        - CN=eric-bss-msg-kafka
        type: simple
      config:
        default.replication.factor: 1
        min.insync.replicas: 1
      listeners:
      - authentication:
          type: tls
        configuration:
          brokerCertChainAndKey:
            certificate: srvcert.crt
            key: srvprivkey.key
            secretName: eric-data-message-bus-kf-test-server-cert-operator-secret
          createBootstrapService: true
        name: siptls
        port: 9093
        tls: true
        type: internal
      - authentication:
          type: tls
        configuration:
          bootstrap: {}
          brokerCertChainAndKey:
            certificate: tls.crt
            key: tls.key
            secretName: eric-bss-msg-kafka-external-kafka-server-cert
          brokers:
          - advertisedHost: eric-data-message-bus-kf-test-kafka-0.hahn157.rnd.gic.ericsson.se
            advertisedPort: 443
            broker: 0
          - advertisedHost: eric-data-message-bus-kf-test-kafka-1.hahn157.rnd.gic.ericsson.se
            advertisedPort: 443
            broker: 1
          - advertisedHost: eric-data-message-bus-kf-test-kafka-2.hahn157.rnd.gic.ericsson.se
            advertisedPort: 443
            broker: 2
          createBootstrapService: true
        name: externaltls
        port: 9094
        tls: true
        type: internal
      logging:
        loggers:
          appender.console.layout.pattern: '{"version": "1.2.0", "timestamp": "%d{yyyy-MM-dd''T''HH:mm:ss.SSSZZZZZ}",
            "severity": "%p{WARN=warning, DEBUG=debug, ERROR=error, FATAL=critical,
            TRACE=debug, INFO=info}", "service_id": "eric-data-messagebus-kf", "message":
            "%enc{%m}{JSON}%enc{%ex{separator(|)}}{JSON}", "metadata": {"container_name":
            "kafka", "pod_name": "$${env:HOSTNAME}"}, "extra_data": {"file": "%file",
            "line": "%line", "mbkf_cluster_name": "eric-data-message-bus-kf-test"}}%n'
          appender.console.layout.type: PatternLayout
          appender.console.name: consoleLogger
          appender.console.target: SYSTEM_OUT
          appender.console.type: Console
          logger.kafka.additivity: "false"
          logger.kafka.appenderRef.console.ref: consoleLogger
          logger.kafka.name: kafka
          logger.kafka.server.DynamicConfigManager.additivity: "false"
          logger.kafka.server.DynamicConfigManager.appenderRef.BufferedSocketAppender.ref: streamLogger
          logger.kafka.server.DynamicConfigManager.appenderRef.console.ref: consoleLogger
          rootLogger.additivity: "false"
          rootLogger.appenderRef.console.ref: consoleLogger
        type: inline
      rack:
        topologyKey: topology.kubernetes.io/zone
      replicas: 3
      resources:
        limits:
          cpu: "2"
          memory: 1Gi
        requests:
          cpu: "1"
          memory: 512Mi
      storage:
        class: network-block
        size: 15Gi
        type: persistent-claim
      template:
        bootstrapService:
          ipFamilyPolicy: PreferDualStack
        brokersService:
          ipFamilyPolicy: PreferDualStack
        kafkaContainer:
          env:
          - name: METRICS_COMPONENT
            value: kafka
          - name: METRICS_EXPORTER_ENABLED
            value: "true"
          - name: DATADIR
            value: /var/lib/kafka/data
          - name: PVCNAME
            value: data-eric-data-message-bus-kf-test-kafka
          - name: METRIC_PREFIX
            value: mbkf
          - name: METRIC_PORT
            value: "9087"
          - name: LOG_LEVEL
            value: info
          - name: ENABLE_TLS_METRIC
            value: "true"
          - name: SERVER_CERT_DIR
            value: /opt/kafka/certificates/metrics-server-secret
          - name: SERVER_CERT_NAME
            value: clicert.pem
          - name: SERVER_CERT_KEY
            value: cliprivkey.pem
          - name: SCRAPE_CA_CERT_DIR
            value: /opt/kafka/certificates/eric-pm-server-ca
          - name: SCRAPE_CA_CERT_NAME
            value: ca.crt
          - name: METRICS_JMX_ENABLED
            value: "true"
          - name: METRICS_JMX_PORT
            value: "9404"
          - name: LOG4J_LOG_LEVEL
            value: INFO
          - name: STRIMZI_CUSTOM_KAFKA_LABELS
            value: 'app.kubernetes.io/version: 3.5.0-ha4ca65e3'
          - name: HELMCHART_VERSION
            value: 3.5.0-ha4ca65e3
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            privileged: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true
        pod:
          affinity:
            podAntiAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
              - podAffinityTerm:
                  labelSelector:
                    matchLabels:
                      strimzi.io/kind: Kafka
                      strimzi.io/name: eric-data-message-bus-kf-test-kafka
                  topologyKey: kubernetes.io/hostname
                weight: 100
          imagePullSecrets:
          - name: armdocker
          metadata:
            annotations:
              authorization-enabled: "true"
              external-listener-ca-thumbprint: d7566f6f,d7566f6f
              kafka-listeners: '{listeners-names=siptls,externaltls}'
              mydomain.io/app: msg1
              mydomain.io/instance: msg1-instance
            labels:
              application: msg1
              eric-data-distributed-coordinator-ed-access: "true"
              eric-sec-key-management-access: "true"
              eric-sec-sip-tls-access: "true"
              instance: msg1-instance
          securityContext:
            fsGroup: 10000
            runAsNonRoot: true
          tolerations:
          - effect: NoSchedule
            key: key1
            operator: Equal
            value: value1
          topologySpreadConstraints:
          - labelSelector:
              matchLabels:
                strimzi.io/kind: Kafka
                strimzi.io/name: eric-data-message-bus-kf-test-kafka
            maxSkew: 1
            topologyKey: zone
            whenUnsatisfiable: ScheduleAnyway
      version: 3.5.1
    zookeeper:
      additionalVolumes:
      - mountPath: /opt/kafka/certificates/eric-sec-sip-tls-trusted-root-cert
        name: eric-sec-sip-tls-trusted-root-cert
        targetContainers:
        - zookeeper
        volumeSource:
          secret:
            items:
            - key: ca.crt
              path: ca.crt
            secretName: eric-sec-sip-tls-trusted-root-cert
          type: secret
      - mountPath: /opt/kafka/certificates/metrics-server-secret
        name: metrics-server-cert
        targetContainers:
        - kafka
        - zookeeper
        volumeSource:
          secret:
            items:
            - key: clicert.pem
              path: clicert.pem
            - key: cliprivkey.pem
              path: cliprivkey.pem
            secretName: eric-data-message-bus-kf-metrics-server-secret
          type: secret
      - mountPath: /opt/kafka/certificates/eric-pm-server-ca
        name: eric-pm-server-ca
        targetContainers:
        - kafka
        - zookeeper
        volumeSource:
          secret:
            items:
            - key: client-cacertbundle.pem
              path: ca.crt
            secretName: eric-pm-server-ca
          type: secret
      logging:
        loggers:
          appender.console.filter.1.onMatch: DENY
          appender.console.filter.1.onMismatch: NEUTRAL
          appender.console.filter.1.regex: .*running stat.*
          appender.console.filter.1.type: RegexFilter
          appender.console.layout.pattern: '{"version": "1.2.0", "timestamp": "%d{yyyy-MM-dd''T''HH:mm:ss.SSSZZZZZ}",
            "severity": "%p{WARN=warning, DEBUG=debug, ERROR=error, FATAL=critical,
            TRACE=debug, INFO=info}", "service_id": "eric-data-messagebus-kf", "message":
            "%enc{%m}{JSON}%enc{%ex{separator(|)}}{JSON}", "metadata": {"container_name":
            "zookeeper", "pod_name": "$${env:HOSTNAME}"}, "extra_data": {"file": "%file",
            "line": "%line", "mbkf_cluster_name": "eric-data-message-bus-kf-test"}}%n'
          appender.console.layout.type: PatternLayout
          appender.console.name: consoleLogger
          appender.console.target: SYSTEM_OUT
          appender.console.type: Console
          rootLogger.additivity: "false"
          rootLogger.appenderRef.console.ref: consoleLogger
        type: inline
      replicas: 1
      resources:
        limits:
          cpu: "2"
          memory: 1Gi
        requests:
          cpu: "1"
          memory: 512Mi
      storage:
        class: network-block
        size: 15Gi
        type: persistent-claim
      template:
        clientService:
          ipFamilyPolicy: PreferDualStack
        nodesService:
          ipFamilyPolicy: PreferDualStack
        pod:
          affinity: {}
          imagePullSecrets:
          - name: armdocker
          metadata:
            annotations:
              mydomain.io/app: msg1
              mydomain.io/instance: msg1-instance
            labels:
              application: msg1
              eric-data-distributed-coordinator-ed-access: "true"
              eric-sec-key-management-access: "true"
              eric-sec-sip-tls-access: "true"
              instance: msg1-instance
          securityContext:
            fsGroup: 10000
            runAsNonRoot: true
        zookeeperContainer:
          env:
          - name: METRICS_COMPONENT
            value: zookeeper
          - name: METRICS_EXPORTER_ENABLED
            value: "true"
          - name: DATADIR
            value: /var/lib/zookeeper
          - name: PVCNAME
            value: data-eric-data-message-bus-kf-test-zookeeper
          - name: METRIC_PREFIX
            value: mbkf
          - name: METRIC_PORT
            value: "9087"
          - name: LOG_LEVEL
            value: info
          - name: ENABLE_TLS_METRIC
            value: "true"
          - name: SERVER_CERT_DIR
            value: /opt/kafka/certificates/metrics-server-secret
          - name: SERVER_CERT_NAME
            value: clicert.pem
          - name: SERVER_CERT_KEY
            value: cliprivkey.pem
          - name: SCRAPE_CA_CERT_DIR
            value: /opt/kafka/certificates/eric-pm-server-ca
          - name: SCRAPE_CA_CERT_NAME
            value: ca.crt
          - name: METRICS_JMX_ENABLED
            value: "true"
          - name: METRICS_JMX_PORT
            value: "9404"
          - name: LOG4J_LOG_LEVEL
            value: INFO
          - name: STRIMZI_CUSTOM_ZOOKEEPER_LABELS
            value: 'app.kubernetes.io/version: 3.5.0-ha4ca65e3'
          - name: HELMCHART_VERSION
            value: 3.5.0-ha4ca65e3
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            privileged: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true
  status:
    clusterId: CAVSR2sNS8ynLfltpYt_fA
    conditions:
    - lastTransitionTime: "2024-10-23T13:47:54.216102875Z"
      status: "True"
      type: Ready
    listeners:
    - addresses:
      - host: eric-data-message-bus-kf-test-kafka-bootstrap.saeed-op.svc
        port: 9093
      bootstrapServers: eric-data-message-bus-kf-test-kafka-bootstrap.saeed-op.svc:9093
      certificates:
      - |-
        -----BEGIN CERTIFICATE-----
        MIIDozCCA0mgAwIBAgIUENmCIXSvAnUzPCgRJDbiCq9qvAUwCgYIKoZIzj0EAwIw
        NDEyMDAGA1UEAxMpZXJpYy1zZWMtc2lwLXRscyBJbnRlcm5hbCBJbnRlcm1lZGlh
        dGUgQ0EwHhcNMjQxMDE4MTUxMTEyWhcNMjUwMzE5MTcxMTQyWjAoMSYwJAYDVQQD
        Ex1lcmljLWRhdGEtbWVzc2FnZS1idXMta2YtdGVzdDBZMBMGByqGSM49AgEGCCqG
        SM49AwEHA0IABA6FIkcr5QVtfymuIF2MgEf+ahD6azBC/8D0tSFsFWtFUsKBQvxx
        gM25J5xytrZcSSVcw0jfTb2BpbS3nG4tICajggJDMIICPzAdBgNVHQ4EFgQUlfhe
        dLK/oSbv+cwvtoE7DnhxrVQwHwYDVR0jBBgwFoAUN4UPHUOoOFR5cBuRR4l71gAY
        tR8wXgYIKwYBBQUHAQEEUjBQME4GCCsGAQUFBzAChkJodHRwczovL2VyaWMtc2Vj
        LWtleS1tYW5hZ2VtZW50OjgyMDAvdjEvZXJpYy1zZWMtc2lwLXRscy1zdWItY2Ev
        Y2EwggF2BgNVHREEggFtMIIBaYIdZXJpYy1kYXRhLW1lc3NhZ2UtYnVzLWtmLXRl
        c3SCJmVyaWMtZGF0YS1tZXNzYWdlLWJ1cy1rZi10ZXN0LnNhZWVkLW9wgiplcmlj
        LWRhdGEtbWVzc2FnZS1idXMta2YtdGVzdC5zYWVlZC1vcC5zdmOCOGVyaWMtZGF0
        YS1tZXNzYWdlLWJ1cy1rZi10ZXN0LnNhZWVkLW9wLnN2Yy5jbHVzdGVyLmxvY2Fs
        gi1lcmljLWRhdGEtbWVzc2FnZS1idXMta2YtdGVzdC1rYWZrYS1ib290c3RyYXCC
        OiouZXJpYy1kYXRhLW1lc3NhZ2UtYnVzLWtmLXRlc3Qta2Fma2EtYnJva2Vycy5z
        YWVlZC1vcC5zdmOCNiouZXJpYy1kYXRhLW1lc3NhZ2UtYnVzLWtmLXRlc3Qta2Fm
        a2EtYnJva2Vycy5zYWVlZC1vcIIXY2VydGlmaWVkLXNjcmFwZS10YXJnZXQwDgYD
        VR0PAQH/BAQDAgeAMBMGA1UdJQQMMAoGCCsGAQUFBwMBMAoGCCqGSM49BAMCA0gA
        MEUCIQCpgCc5Q2KVCC4rFXX//BHUm/+NjaUfHzqnaVqWpN9mUQIgFzKLIoCzNImk
        8R+lN6sdzqqR4DVUF05WuU62YcvsRMs=
        -----END CERTIFICATE-----
        -----BEGIN CERTIFICATE-----
        MIICKzCCAdCgAwIBAgIUa5RyaG1VGoi9jYkBRYG/8UfoRgMwCgYIKoZIzj0EAwIw
        LDEqMCgGA1UEAxMhZXJpYy1zZWMtc2lwLXRscyBJbnRlcm5hbCBSb290IENBMCAX
        DTI0MTAxODE0MDcxNVoYDzIxMjQwOTI0MTQwNzQ1WjA0MTIwMAYDVQQDEyllcmlj
        LXNlYy1zaXAtdGxzIEludGVybmFsIEludGVybWVkaWF0ZSBDQTBZMBMGByqGSM49
        AgEGCCqGSM49AwEHA0IABBnmmnDQvL7SlUP5TcrFtKNjK8jw7W+meHlJ0YivDoRP
        +NJ1iOJclcV3lAP60cRfs2UP713a17o0d4yJixAJiVyjgcUwgcIwDgYDVR0PAQH/
        BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFDeFDx1DqDhUeXAbkUeJ
        e9YAGLUfMB8GA1UdIwQYMBaAFPp0Mq4fIIOyw4ocI11gBVLENt5KMF8GCCsGAQUF
        BwEBBFMwUTBPBggrBgEFBQcwAoZDaHR0cHM6Ly9lcmljLXNlYy1rZXktbWFuYWdl
        bWVudDo4MjAwL3YxL2VyaWMtc2VjLXNpcC10bHMtcm9vdC1jYS9jYTAKBggqhkjO
        PQQDAgNJADBGAiEA8uqrhtyOyodbmVhfLiJ+MiZA9b4WKdFiD0MYPxNaa7ICIQD1
        YaduAL6qq9KrAR4R6gzrr0tMfTid4omSl613MNZKWA==
        -----END CERTIFICATE-----
      name: siptls
      type: siptls
    - addresses:
      - host: eric-data-message-bus-kf-test-kafka-bootstrap.saeed-op.svc
        port: 9094
      bootstrapServers: eric-data-message-bus-kf-test-kafka-bootstrap.saeed-op.svc:9094
      certificates:
      - |
        -----BEGIN CERTIFICATE-----
        MIIEGTCCAgGgAwIBAgICA+kwDQYJKoZIhvcNAQELBQAwEDEOMAwGA1UEAxMFU1JW
        Q0EwHhcNMjQxMDE4MTQ0MzI1WhcNMjUxMDI4MTU0MzI1WjAmMSQwIgYDVQQDDBsq
        LmthZmthLWV4dC50ZXN0ZXhhbXBsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IB
        DwAwggEKAoIBAQCpaPSubgm69wYiDyUsWXjSOiHNikiAukwOoO5hzAiDHhDV1kmL
        tWuAUXHPu1Ko8g4Vo8xsYE1NVFT35AanoTqzDaIwOz25tJ5sxy3IwyGhZSQlensL
        MyDrjM5+6iuFxRHGVFecn3Kw+YFZgovWqJ84jVS0CIwgMc90br0obVmR+4xmm8RF
        kT6qroNJlTq3/rUeLuby13Rgwsfgyyuvf60HYXorUcjgClLf3CkVl12HF+5/Sw3Z
        Dh0ANRPq1jDIjQIxn12nkTlt1Ua9aeYOONGCIvGKjqfUNsZwGyvnaXCsb8E4qy1a
        bflbgDLexRkB2ZKAtuHtTFBA2qy5awv3JX7xAgMBAAGjZzBlMA4GA1UdDwEB/wQE
        AwIF4DATBgNVHSUEDDAKBggrBgEFBQcDATAdBgNVHQ4EFgQUkmBeHnzQRtdk2evX
        D35/6qRGfdQwHwYDVR0jBBgwFoAUa+qzLNG5SOLi27PQTRllkrlHJHYwDQYJKoZI
        hvcNAQELBQADggIBAGtgAvTz1KuAnDYNHbrbpbwkjIygQZb50lVbjIRmcOA3ntq8
        zhE3cPU2OBQdED7OSMzEncqsHp0/1K9OcqdEaR9FI3rAPsCJP2id/Pnwbj1jLf9F
        EYVdgOGjUbfIvSpLt7TQiWon9s+2ZI2K4h2cggtQ2FH00LbT+EKMQYXBB8hSuQqc
        Yu/CXouSC1Wq2p88F+CL7bLXs7oy7eCWSnYo9SuzXM5mGrFuq0Aqrc4tCEH8My6d
        EGE8coisWDOGG0mUR3JJNWfY7j4aVtt+QQLHwpP4OP4eLMjYBpkQ3PbZHlNgi6QX
        PefFM7BxYbyfgJpOiKaWHJi8skG02YpHWFv7T0X1xKw/OzGdOyRZFYHTBzYCcuzW
        jWTpmujURmicCrwmPivkex7YZTCsy9QQxbwX2ZoCgTiM7gt72eYvBLzbGljC8HK6
        un4sK46qo+qt0R0hjiCEyLcvJG2aD7huTGXaCALDwp/BbVEd7KbWM+Y232+9xDzp
        6xY5I86E4JuxTc9M2jMrNn4kFL5pUP6FlNPVZLxNLpyh0UV9hh7NIJVU/uUPHxON
        cgL9uGOHZ9wGxTgK6yD/Vi12XvM6kjVJGOd/X75RmBcBTQXHKggrCv648WWArNZ+
        WfyAuy8YnXtJHmAjdVCML7maE5YuChy5VZsB5blbuGFF7cbG3shZnrPwxLiO
        -----END CERTIFICATE-----
      name: externaltls
      type: externaltls
    observedGeneration: 16
kind: List
metadata:
  resourceVersion: ""

This is the config and i am going to delete "eric-bss-msg-kafka-external-kafka-client-ca" secret and re-create it again, that will cause the pods to restart but when i check the events i am expecting "CaCertRenewed" but it is still "PodHasOldRevision"

[scholzj]

This is the config and i am going to delete "eric-bss-msg-kafka-external-kafka-client-ca" secret and re-create it again, that will cause the pods to restart but when i check the events i am expecting "CaCertRenewed" but it is still "PodHasOldRevision"

Wait, why would you do that?

• That is not a CA renewal
• And you are also not using external CA but Strimzi generated CAs based on the configuration you shared

If you want to use your own CA or trigger a renewal, you should follow the docs. Just by going in and deleting some secret you are just doing some random disaster recovery testing.