adding 4th broker crashing in 0.38 operator #9685

Veljen · 2024-02-14T10:10:23Z

Veljen
Feb 14, 2024

Hi Team

I tried to add 4th broker to the kafka and its not creating and its giving below error.

poc-strimizi-kafka-0                           1/1     Running            0             48m
poc-strimizi-kafka-1                           1/1     Running            0             48m
poc-strimizi-kafka-2                           1/1     Running            0             48m
poc-strimizi-kafka-3                           0/1     CrashLoopBackOff   5 (49s ago)   6m7s

2024-02-14 10:04:38 INFO  AbstractOperator:265 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): Kafka poc-strimizi-2 will be checked for creation or modification
2024-02-14 10:04:38 INFO  AbstractOperator:265 - Reconciliation #327(timer) Kafka(test/test-strimzi): Kafka test-strimzi will be checked for creation or modification
2024-02-14 10:04:43 INFO  KafkaRoller:355 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-0/0 is up-to-date due to ForceableProblem: Pod poc-strimizi-2-kafka-0 is controller and there are other pods to verify. Non-controller pods will be verified first, retrying after at least 250ms
2024-02-14 10:04:44 INFO  KafkaAvailability:135 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): mm2-offset-syncs.poc-strimzi-1.internal/0 will be under-replicated (ISR={1}, replicas=[1,0,2], min.insync.replicas=1) if broker 1 is restarted.
2024-02-14 10:04:44 INFO  KafkaRoller:355 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-1/1 is up-to-date due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod poc-strimizi-2-kafka-1 cannot be updated right now., retrying after at least 250ms
2024-02-14 10:04:44 WARN  AbstractModel:286 - Reconciliation #327(timer) Kafka(test/test-strimzi): Anomaly goals contained goals which are not in the configured default goals. Anomaly goals have been changed to match the specified default goals.
2024-02-14 10:04:45 INFO  KafkaRoller:355 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-0/0 is up-to-date due to ForceableProblem: Pod poc-strimizi-2-kafka-0 is controller and there are other pods to verify. Non-controller pods will be verified first, retrying after at least 500ms
2024-02-14 10:04:45 INFO  AbstractOperator:537 - Reconciliation #327(timer) Kafka(test/test-strimzi): reconciled
2024-02-14 10:04:46 INFO  KafkaAvailability:135 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): mm2-offset-syncs.poc-strimzi-1.internal/0 will be under-replicated (ISR={1}, replicas=[1,0,2], min.insync.replicas=1) if broker 1 is restarted.
2024-02-14 10:04:46 INFO  KafkaRoller:355 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-1/1 is up-to-date due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod poc-strimizi-2-kafka-1 cannot be updated right now., retrying after at least 500ms
2024-02-14 10:04:47 INFO  KafkaRoller:355 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-0/0 is up-to-date due to ForceableProblem: Pod poc-strimizi-2-kafka-0 is controller and there are other pods to verify. Non-controller pods will be verified first, retrying after at least 1000ms
2024-02-14 10:04:48 INFO  KafkaAvailability:135 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): mm2-offset-syncs.poc-strimzi-1.internal/0 will be under-replicated (ISR={1}, replicas=[1,0,2], min.insync.replicas=1) if broker 1 is restarted.
2024-02-14 10:04:48 INFO  KafkaRoller:355 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-1/1 is up-to-date due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod poc-strimizi-2-kafka-1 cannot be updated right now., retrying after at least 1000ms
2024-02-14 10:04:49 INFO  KafkaRoller:355 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-0/0 is up-to-date due to ForceableProblem: Pod poc-strimizi-2-kafka-0 is controller and there are other pods to verify. Non-controller pods will be verified first, retrying after at least 2000ms
2024-02-14 10:04:50 INFO  KafkaAvailability:135 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): mm2-offset-syncs.poc-strimzi-1.internal/0 will be under-replicated (ISR={1}, replicas=[1,0,2], min.insync.replicas=1) if broker 1 is restarted.
2024-02-14 10:04:50 INFO  KafkaRoller:355 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-1/1 is up-to-date due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod poc-strimizi-2-kafka-1 cannot be updated right now., retrying after at least 2000ms
2024-02-14 10:04:51 INFO  KafkaRoller:355 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-0/0 is up-to-date due to ForceableProblem: Pod poc-strimizi-2-kafka-0 is controller and there are other pods to verify. Non-controller pods will be verified first, retrying after at least 4000ms
2024-02-14 10:04:53 INFO  KafkaAvailability:135 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): mm2-offset-syncs.poc-strimzi-1.internal/0 will be under-replicated (ISR={1}, replicas=[1,0,2], min.insync.replicas=1) if broker 1 is restarted.
2024-02-14 10:04:53 INFO  KafkaRoller:355 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-1/1 is up-to-date due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod poc-strimizi-2-kafka-1 cannot be updated right now., retrying after at least 4000ms
2024-02-14 10:04:56 INFO  KafkaRoller:355 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-0/0 is up-to-date due to ForceableProblem: Pod poc-strimizi-2-kafka-0 is controller and there are other pods to verify. Non-controller pods will be verified first, retrying after at least 8000ms
2024-02-14 10:04:57 INFO  KafkaAvailability:135 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): mm2-offset-syncs.poc-strimzi-1.internal/0 will be under-replicated (ISR={1}, replicas=[1,0,2], min.insync.replicas=1) if broker 1 is restarted.
2024-02-14 10:04:57 INFO  KafkaRoller:355 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-1/1 is up-to-date due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod poc-strimizi-2-kafka-1 cannot be updated right now., retrying after at least 8000ms
2024-02-14 10:05:04 INFO  StrimziPodSetController:378 - Reconciliation #328(watch) StrimziPodSet(strimzi-kafka-1/poc-strimizi-kafka): StrimziPodSet will be reconciled
2024-02-14 10:05:04 INFO  StrimziPodSetController:414 - Reconciliation #328(watch) StrimziPodSet(strimzi-kafka-1/poc-strimizi-kafka): reconciled
2024-02-14 10:05:04 INFO  KafkaRoller:355 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-0/0 is up-to-date due to ForceableProblem: Pod poc-strimizi-2-kafka-0 is controller and there are other pods to verify. Non-controller pods will be verified first, retrying after at least 16000ms
2024-02-14 10:05:06 INFO  KafkaAvailability:135 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): mm2-offset-syncs.poc-strimzi-1.internal/0 will be under-replicated (ISR={1}, replicas=[1,0,2], min.insync.replicas=1) if broker 1 is restarted.
2024-02-14 10:05:06 INFO  KafkaRoller:355 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-1/1 is up-to-date due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod poc-strimizi-2-kafka-1 cannot be updated right now., retrying after at least 16000ms
2024-02-14 10:05:21 INFO  KafkaRoller:355 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-0/0 is up-to-date due to ForceableProblem: Pod poc-strimizi-2-kafka-0 is controller and there are other pods to verify. Non-controller pods will be verified first, retrying after at least 32000ms
2024-02-14 10:05:23 INFO  KafkaAvailability:135 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): mm2-offset-syncs.poc-strimzi-1.internal/0 will be under-replicated (ISR={1}, replicas=[1,0,2], min.insync.replicas=1) if broker 1 is restarted.
2024-02-14 10:05:23 INFO  KafkaRoller:355 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-1/1 is up-to-date due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod poc-strimizi-2-kafka-1 cannot be updated right now., retrying after at least 32000ms
2024-02-14 10:05:38 INFO  AbstractOperator:400 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): Reconciliation is in progress
2024-02-14 10:05:38 INFO  AbstractOperator:400 - Reconciliation #309(timer) Kafka(strimzi-kafka-1/poc-strimizi): Reconciliation is in progress
2024-02-14 10:05:44 INFO  StrimziPodSetController:378 - Reconciliation #329(watch) StrimziPodSet(strimzi-kafka-1/poc-strimizi-kafka): StrimziPodSet will be reconciled
2024-02-14 10:05:44 INFO  StrimziPodSetController:414 - Reconciliation #329(watch) StrimziPodSet(strimzi-kafka-1/poc-strimizi-kafka): reconciled
2024-02-14 10:05:54 INFO  KafkaRoller:355 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-0/0 is up-to-date due to ForceableProblem: Pod poc-strimizi-2-kafka-0 is controller and there are other pods to verify. Non-controller pods will be verified first, retrying after at least 64000ms
2024-02-14 10:05:56 INFO  KafkaAvailability:135 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): mm2-offset-syncs.poc-strimzi-1.internal/0 will be under-replicated (ISR={1}, replicas=[1,0,2], min.insync.replicas=1) if broker 1 is restarted.
2024-02-14 10:05:56 INFO  KafkaRoller:355 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-1/1 is up-to-date due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod poc-strimizi-2-kafka-1 cannot be updated right now., retrying after at least 64000ms
2024-02-14 10:06:22 INFO  StrimziPodSetController:378 - Reconciliation #330(watch) StrimziPodSet(strimzi-kafka-1/poc-strimizi-kafka): StrimziPodSet will be reconciled
2024-02-14 10:06:22 INFO  StrimziPodSetController:414 - Reconciliation #330(watch) StrimziPodSet(strimzi-kafka-1/poc-strimizi-kafka): reconciled
2024-02-14 10:06:32 INFO  ClusterOperator:142 - Triggering periodic reconciliation for namespace kafkaoperator
2024-02-14 10:06:38 INFO  ClusterOperator:142 - Triggering periodic reconciliation for namespace strimzi-kafka-2
2024-02-14 10:06:38 INFO  AbstractOperator:400 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): Reconciliation is in progress
2024-02-14 10:06:38 INFO  ClusterOperator:142 - Triggering periodic reconciliation for namespace strimzi-kafka-1
2024-02-14 10:06:38 INFO  AbstractOperator:400 - Reconciliation #309(timer) Kafka(strimzi-kafka-1/poc-strimizi): Reconciliation is in progress
2024-02-14 10:06:38 INFO  ClusterOperator:142 - Triggering periodic reconciliation for namespace test
2024-02-14 10:06:38 INFO  AbstractOperator:265 - Reconciliation #332(timer) Kafka(test/test-strimzi): Kafka test-strimzi will be checked for creation or modification
2024-02-14 10:06:43 WARN  AbstractModel:286 - Reconciliation #332(timer) Kafka(test/test-strimzi): Anomaly goals contained goals which are not in the configured default goals. Anomaly goals have been changed to match the specified default goals.
2024-02-14 10:06:44 INFO  AbstractOperator:537 - Reconciliation #332(timer) Kafka(test/test-strimzi): reconciled
2024-02-14 10:06:58 INFO  KafkaAvailability:127 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): __strimzi_store_topic/0 is already under-replicated (ISR={2}, replicas=[1,2,0], min.insync.replicas=2); broker 0 has a replica, so should not be restarted right now (it might be first to catch up).
2024-02-14 10:06:58 WARN  KafkaRoller:460 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): Pod poc-strimizi-2-kafka-0/0 can't be safely force-rolled; original error:
2024-02-14 10:06:58 INFO  KafkaRoller:348 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): Could not verify pod poc-strimizi-2-kafka-0/0 is up-to-date, giving up after 10 attempts. Total delay between attempts 127750ms
io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod poc-strimizi-2-kafka-0 is controller and there are other pods to verify. Non-controller pods will be verified first
        at io.strimzi.operator.cluster.operator.resource.KafkaRoller.restartIfNecessary(KafkaRoller.java:425) ~[io.strimzi.cluster-operator-0.38.0.jar:0.38.0]
        at io.strimzi.operator.cluster.operator.resource.KafkaRoller.lambda$schedule$6(KafkaRoller.java:333) ~[io.strimzi.cluster-operator-0.38.0.jar:0.38.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-02-14 10:07:01 INFO  KafkaAvailability:135 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): mm2-offset-syncs.poc-strimzi-1.internal/0 will be under-replicated (ISR={1}, replicas=[1,0,2], min.insync.replicas=1) if broker 1 is restarted.
2024-02-14 10:07:01 INFO  KafkaRoller:348 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): Could not verify pod poc-strimizi-2-kafka-1/1 is up-to-date, giving up after 10 attempts. Total delay between attempts 127750ms
io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod poc-strimizi-2-kafka-1 cannot be updated right now.
        at io.strimzi.operator.cluster.operator.resource.KafkaRoller.restartIfNecessary(KafkaRoller.java:437) ~[io.strimzi.cluster-operator-0.38.0.jar:0.38.0]
        at io.strimzi.operator.cluster.operator.resource.KafkaRoller.lambda$schedule$6(KafkaRoller.java:333) ~[io.strimzi.cluster-operator-0.38.0.jar:0.38.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-02-14 10:07:01 ERROR AbstractOperator:284 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): createOrUpdate failed
io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod poc-strimizi-2-kafka-0 is controller and there are other pods to verify. Non-controller pods will be verified first
        at io.strimzi.operator.cluster.operator.resource.KafkaRoller.restartIfNecessary(KafkaRoller.java:425) ~[io.strimzi.cluster-operator-0.38.0.jar:0.38.0]
        at io.strimzi.operator.cluster.operator.resource.KafkaRoller.lambda$schedule$6(KafkaRoller.java:333) ~[io.strimzi.cluster-operator-0.38.0.jar:0.38.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-02-14 10:07:01 WARN  AbstractOperator:557 - Reconciliation #325(timer) Kafka(strimzi-kafka-2/poc-strimizi-2): Failed to reconcile
io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod poc-strimizi-2-kafka-0 is controller and there are other pods to verify. Non-controller pods will be verified first
        at io.strimzi.operator.cluster.operator.resource.KafkaRoller.restartIfNecessary(KafkaRoller.java:425) ~[io.strimzi.cluster-operator-0.38.0.jar:0.38.0]
        at io.strimzi.operator.cluster.operator.resource.KafkaRoller.lambda$schedule$6(KafkaRoller.java:333) ~[io.strimzi.cluster-operator-0.38.0.jar:0.38.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-02-14 10:07:04 INFO  StrimziPodSetController:378 - Reconciliation #334(watch) StrimziPodSet(strimzi-kafka-1/poc-strimizi-kafka): StrimziPodSet will be reconciled
2024-02-14 10:07:04 INFO  StrimziPodSetController:414 - Reconciliation #334(watch) StrimziPodSet(strimzi-kafka-1/poc-strimizi-kafka): reconciled
2024-02-14 10:07:38 INFO  AbstractOperator:400 - Reconciliation #309(timer) Kafka(strimzi-kafka-1/poc-strimizi): Reconciliation is in progress
2024-02-14 10:07:46 INFO  StrimziPodSetController:378 - Reconciliation #335(watch) StrimziPodSet(strimzi-kafka-1/poc-strimizi-kafka): StrimziPodSet will be reconciled
2024-02-14 10:07:46 INFO  StrimziPodSetController:414 - Reconciliation #335(watch) StrimziPodSet(strimzi-kafka-1/poc-strimizi-kafka): reconciled

scholzj · 2024-02-14T10:23:56Z

scholzj
Feb 14, 2024
Maintainer

The log you shared suggests that you are trying to roll the cluster but some brokers are not in-sync and cannot be rolled. It is not clear from it why is it rolling the cluster or why is one of the pods crashlooping. You would need to share more information - full configuration (Kafka CR), full log, log from the brokers etc.

0 replies

Veljen · 2024-02-14T10:41:32Z

Veljen
Feb 14, 2024
Author

kafka cR

apiVersion: v1
items:
- apiVersion: kafka.strimzi.io/v1beta2
  kind: Kafka
  metadata:
    creationTimestamp: "2024-02-14T09:16:46Z"
    generation: 3
    name: poc-strimizi
    namespace: strimzi-kafka-1
    resourceVersion: "9873600"
    uid: cfb15b70-9cfc-49fa-918f-7e80dc2d4cb3
  spec:
    clientsCa:
      generateCertificateAuthority: true
      renewalDays: 30
      validityDays: 3650
    clusterCa:
      generateCertificateAuthority: true
      renewalDays: 30
      validityDays: 3650
    cruiseControl:
      config:
        default.goals: |
          com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal,
          com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,
          com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,
          com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderReplicaDistributionGoal
        goals: |
          com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal,
          com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,
          com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,
          com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderReplicaDistributionGoal
        hard.goals: |
          com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal,
          com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,
          com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,
          com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderReplicaDistributionGoal
        webserver.http.cors.enabled: "true"
        webserver.http.cors.exposeheaders: User-Task-ID,Content-Type
        webserver.http.cors.origin: '*'
        webserver.security.enable: "false"
      image: vel123/cruise-control-ui:latest
      resources:
        limits:
          cpu: 1
          memory: 1Gi
        requests:
          cpu: 500m
          memory: 512Mi
      template:
        pod:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: app
                    operator: In
                    values:
                    - kafka
          metadata:
            labels:
              app: kafka
              environment: production
          tolerations:
          - effect: NoSchedule
            key: app
            operator: Equal
            value: kafka
    entityOperator:
      template:
        pod:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: app
                    operator: In
                    values:
                    - kafka
          metadata:
            annotations:
              newrelic.io/scrape: "true"
            labels:
              kafka-metrics: "true"
          tolerations:
          - effect: NoSchedule
            key: app
            operator: Equal
            value: kafka
      topicOperator: {}
      userOperator: {}
    kafka:
      config:
        default.replication.factor: 3
        inter.broker.protocol.version: "3.6"
        min.insync.replicas: 2
        offsets.topic.replication.factor: 3
        transaction.state.log.min.isr: 2
        transaction.state.log.replication.factor: 3
      listeners:
      - name: plain
        port: 9092
        tls: false
        type: internal
      - name: tls
        port: 9093
        tls: true
        type: internal
      - configuration:
          bootstrap:
            annotations:
              service.beta.kubernetes.io/azure-load-balancer-internal: "true"
          brokers:
          - annotations:
              service.beta.kubernetes.io/azure-load-balancer-internal: "true"
            broker: 0
          - annotations:
              service.beta.kubernetes.io/azure-load-balancer-internal: "true"
            broker: 1
          - annotations:
              service.beta.kubernetes.io/azure-load-balancer-internal: "true"
            broker: 2
          - annotations:
              service.beta.kubernetes.io/azure-load-balancer-internal: "true"
            broker: 3
        name: external
        port: 9094
        tls: false
        type: loadbalancer
      metricsConfig:
        type: jmxPrometheusExporter
        valueFrom:
          configMapKeyRef:
            key: kafka-metrics-config.yml
            name: kafka-metrics
      replicas: 4
      storage:
        type: jbod
        volumes:
        - deleteClaim: false
          id: 0
          size: 100Gi
          type: persistent-claim
      template:
        pod:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: app
                    operator: In
                    values:
                    - kafka
          metadata:
            annotations:
              newrelic.io/scrape: "true"
          tolerations:
          - key: app
            operator: Equal
            value: kafka
      version: 3.6.0
    kafkaExporter:
      groupRegex: .*
      template:
        pod:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: app
                    operator: In
                    values:
                    - kafka
          metadata:
            annotations:
              prometheus.io/scrape: "true"
            labels:
              kafka-metrics: "true"
          tolerations:
          - key: app
            operator: Equal
            value: kafka
      topicRegex: .*
    zookeeper:
      replicas: 3
      storage:
        deleteClaim: false
        size: 100Gi
        type: persistent-claim
      template:
        pod:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: app
                    operator: In
                    values:
                    - kafka
          tolerations:
          - key: app
            operator: Equal
            value: kafka

I have deleted all pods, now all pods are crashing.

poc-strimizi-kafka-0                           0/1     CrashLoopBackOff   5 (31s ago)   4m51s
poc-strimizi-kafka-1                           0/1     CrashLoopBackOff   5 (52s ago)   5m7s
poc-strimizi-kafka-2                           0/1     Running            5 (22s ago)   4m55s
poc-strimizi-kafka-3                           0/1     CrashLoopBackOff   5 (64s ago)   5m34s

Its repeatedly crash - broker logs

removed directory '/tmp/hsperfdata_kafka'
removed '/tmp/kafka/clients.truststore.p12'
removed '/tmp/kafka/cluster.keystore.p12'
removed '/tmp/kafka/cluster.truststore.p12'
removed directory '/tmp/kafka'
removed '/tmp/strimzi.properties'
STRIMZI_BROKER_ID=3
Preparing truststore for replication listener
Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore /tmp/kafka/cluster.truststore.p12 with alias ca
Certificate was added to keystore
Preparing truststore for replication listener is complete
Looking for the right CA
Found the right CA: /opt/kafka/cluster-ca-certs/ca.crt
Preparing keystore for replication and clienttls listener
Preparing keystore for replication and clienttls listener is complete
Preparing truststore for client authentication
Adding /opt/kafka/client-ca-certs/ca.crt to truststore /tmp/kafka/clients.truststore.p12 with alias ca
Certificate was added to keystore
Preparing truststore for client authentication is complete
Starting Kafka with configuration:
##############################
##############################
# This file is automatically generated by the Strimzi Cluster Operator
# Any changes to this file will be ignored and overwritten!
##############################
##############################

##########
# Node / Broker ID
##########
broker.id=3
node.id=3

##########
# Kafka message logs configuration
##########
log.dirs=/var/lib/kafka/data-0/kafka-log3

##########
# Control Plane listener
##########
listener.name.controlplane-9090.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
listener.name.controlplane-9090.ssl.keystore.password=[hidden]
listener.name.controlplane-9090.ssl.keystore.type=PKCS12
listener.name.controlplane-9090.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
listener.name.controlplane-9090.ssl.truststore.password=[hidden]
listener.name.controlplane-9090.ssl.truststore.type=PKCS12
listener.name.controlplane-9090.ssl.client.auth=required

##########
# Replication listener
##########
listener.name.replication-9091.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
listener.name.replication-9091.ssl.keystore.password=[hidden]
listener.name.replication-9091.ssl.keystore.type=PKCS12
listener.name.replication-9091.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
listener.name.replication-9091.ssl.truststore.password=[hidden]
listener.name.replication-9091.ssl.truststore.type=PKCS12
listener.name.replication-9091.ssl.client.auth=required

##########
# Listener configuration: PLAIN-9092
##########

##########
# Listener configuration: TLS-9093
##########
listener.name.tls-9093.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
listener.name.tls-9093.ssl.keystore.password=[hidden]
listener.name.tls-9093.ssl.keystore.type=PKCS12


##########
# Listener configuration: EXTERNAL-9094
##########

##########
# Common listener configuration
##########
listener.security.protocol.map=CONTROLPLANE-9090:SSL,REPLICATION-9091:SSL,PLAIN-9092:PLAINTEXT,TLS-9093:SSL,EXTERNAL-9094:PLAINTEXT
listeners=CONTROLPLANE-9090://0.0.0.0:9090,REPLICATION-9091://0.0.0.0:9091,PLAIN-9092://0.0.0.0:9092,TLS-9093://0.0.0.0:9093,EXTERNAL-9094://0.0.0.0:9094
advertised.listeners=CONTROLPLANE-9090://poc-strimizi-kafka-3.poc-strimizi-kafka-brokers.strimzi-kafka-1.svc:9090,REPLICATION-9091://poc-strimizi-kafka-3.poc-strimizi-kafka-brokers.strimzi-kafka-1.svc:9091,PLAIN-9092://poc-strimizi-kafka-3.poc-strimizi-kafka-brokers.strimzi-kafka-1.svc:9092,TLS-9093://poc-strimizi-kafka-3.poc-strimizi-kafka-brokers.strimzi-kafka-1.svc:9093,EXTERNAL-9094://10.88.4.44:9094
inter.broker.listener.name=REPLICATION-9091
control.plane.listener.name=CONTROLPLANE-9090
sasl.enabled.mechanisms=
ssl.endpoint.identification.algorithm=HTTPS

##########
# Cruise Control configuration
##########
cruise.control.metrics.topic=strimzi.cruisecontrol.metrics
cruise.control.metrics.reporter.ssl.endpoint.identification.algorithm=HTTPS
cruise.control.metrics.reporter.bootstrap.servers=poc-strimizi-kafka-brokers:9091
cruise.control.metrics.reporter.security.protocol=SSL
cruise.control.metrics.reporter.ssl.keystore.type=PKCS12
cruise.control.metrics.reporter.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
cruise.control.metrics.reporter.ssl.keystore.password=[hidden]
cruise.control.metrics.reporter.ssl.truststore.type=PKCS12
cruise.control.metrics.reporter.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
cruise.control.metrics.reporter.ssl.truststore.password=[hidden]
cruise.control.metrics.topic.auto.create=true
cruise.control.metrics.topic.num.partitions=1
cruise.control.metrics.topic.replication.factor=3
cruise.control.metrics.topic.min.insync.replicas=1

##########
# User provided configuration
##########
default.replication.factor=3
inter.broker.protocol.version=3.6
min.insync.replicas=2
offsets.topic.replication.factor=3
transaction.state.log.min.isr=2
transaction.state.log.replication.factor=3
log.message.format.version=3.6
metric.reporters=com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter


##########
# Zookeeper
##########
zookeeper.connect=poc-strimizi-zookeeper-client:2181
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
zookeeper.ssl.client.enable=true
zookeeper.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
zookeeper.ssl.keystore.password=[hidden]
zookeeper.ssl.keystore.type=PKCS12
zookeeper.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
zookeeper.ssl.truststore.password=[hidden]
zookeeper.ssl.truststore.type=PKCS12
+ exec /usr/bin/tini -w -e 143 -- /opt/kafka/bin/kafka-server-start.sh /tmp/strimzi.properties

operator logs

Auto-detected KUBERNETES_SERVICE_DNS_DOMAIN: cluster.local
+ exec /usr/bin/tini -w -e 143 -- java -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom -XX:+ExitOnOutOfMemoryError -XX:MinRAMPercentage=15 -XX:MaxRAMPercentage=20 -classpath :lib/io.strimzi.cluster-operator-0.38.0.jar:lib/io.strimzi.api-0.38.0.jar:lib/io.fabric8.kubernetes-model-apiextensions-6.9.0.jar:lib/io.strimzi.config-model-0.38.0.jar:lib/io.strimzi.operator-common-0.38.0.jar:lib/io.strimzi.crd-annotations-0.38.0.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar:lib/org.eclipse.jetty.jetty-server-9.4.53.v20231009.jar:lib/org.eclipse.jetty.jetty-http-9.4.53.v20231009.jar:lib/org.eclipse.jetty.jetty-io-9.4.53.v20231009.jar:lib/javax.servlet.javax.servlet-api-3.1.0.jar:lib/io.fabric8.openshift-client-6.9.0.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.fabric8.openshift-client-api-6.9.0.jar:lib/io.fabric8.openshift-model-clusterautoscaling-6.9.0.jar:lib/io.fabric8.openshift-model-operator-6.9.0.jar:lib/io.fabric8.openshift-model-operatorhub-6.9.0.jar:lib/io.fabric8.openshift-model-machine-6.9.0.jar:lib/io.fabric8.openshift-model-whereabouts-6.9.0.jar:lib/io.fabric8.openshift-model-monitoring-6.9.0.jar:lib/io.fabric8.openshift-model-storageversionmigrator-6.9.0.jar:lib/io.fabric8.openshift-model-tuned-6.9.0.jar:lib/io.fabric8.openshift-model-console-6.9.0.jar:lib/io.fabric8.openshift-model-config-6.9.0.jar:lib/io.fabric8.openshift-model-machineconfig-6.9.0.jar:lib/io.fabric8.openshift-model-miscellaneous-6.9.0.jar:lib/io.fabric8.openshift-model-hive-6.9.0.jar:lib/io.fabric8.openshift-model-installer-6.9.0.jar:lib/io.fabric8.kubernetes-client-6.9.0.jar:lib/io.fabric8.kubernetes-httpclient-jdk-6.9.0.jar:lib/io.fabric8.kubernetes-client-api-6.9.0.jar:lib/io.fabric8.kubernetes-model-gatewayapi-6.9.0.jar:lib/io.fabric8.kubernetes-model-resource-6.9.0.jar:lib/io.fabric8.kubernetes-model-admissionregistration-6.9.0.jar:lib/io.fabric8.kubernetes-model-autoscaling-6.9.0.jar:lib/io.fabric8.kubernetes-model-batch-6.9.0.jar:lib/io.fabric8.kubernetes-model-certificates-6.9.0.jar:lib/io.fabric8.kubernetes-model-discovery-6.9.0.jar:lib/io.fabric8.kubernetes-model-extensions-6.9.0.jar:lib/io.fabric8.kubernetes-model-flowcontrol-6.9.0.jar:lib/io.fabric8.kubernetes-model-metrics-6.9.0.jar:lib/io.fabric8.kubernetes-model-scheduling-6.9.0.jar:lib/io.fabric8.kubernetes-model-node-6.9.0.jar:lib/org.slf4j.slf4j-api-1.7.36.jar:lib/org.snakeyaml.snakeyaml-engine-2.7.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.15.2.jar:lib/io.fabric8.kubernetes-model-core-6.9.0.jar:lib/io.fabric8.kubernetes-model-common-6.9.0.jar:lib/io.fabric8.kubernetes-model-events-6.9.0.jar:lib/io.fabric8.kubernetes-model-policy-6.9.0.jar:lib/io.fabric8.kubernetes-model-rbac-6.9.0.jar:lib/io.fabric8.kubernetes-model-apps-6.9.0.jar:lib/io.fabric8.kubernetes-model-storageclass-6.9.0.jar:lib/io.fabric8.kubernetes-model-networking-6.9.0.jar:lib/io.fabric8.kubernetes-model-coordination-6.9.0.jar:lib/io.fabric8.openshift-model-6.9.0.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.15.2.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.15.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.15.2.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.15.2.jar:lib/org.yaml.snakeyaml-2.0.jar:lib/io.vertx.vertx-core-4.4.6.jar:lib/io.netty.netty-common-4.1.100.Final.jar:lib/io.netty.netty-buffer-4.1.100.Final.jar:lib/io.netty.netty-handler-4.1.100.Final.jar:lib/io.netty.netty-handler-proxy-4.1.100.Final.jar:lib/io.netty.netty-codec-http-4.1.100.Final.jar:lib/io.netty.netty-codec-http2-4.1.100.Final.jar:lib/io.netty.netty-resolver-4.1.100.Final.jar:lib/io.netty.netty-resolver-dns-4.1.100.Final.jar:lib/io.netty.netty-codec-dns-4.1.100.Final.jar:lib/io.strimzi.certificate-manager-0.38.0.jar:lib/org.apache.logging.log4j.log4j-core-2.17.2.jar:lib/org.apache.logging.log4j.log4j-api-2.17.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar:lib/io.strimzi.kafka-oauth-server-0.14.0.jar:lib/io.strimzi.kafka-oauth-server-plain-0.14.0.jar:lib/io.strimzi.kafka-oauth-client-0.14.0.jar:lib/io.strimzi.kafka-oauth-common-0.14.0.jar:lib/com.nimbusds.nimbus-jose-jwt-9.31.jar:lib/com.github.stephenc.jcip.jcip-annotations-1.0-1.jar:lib/com.jayway.jsonpath.json-path-2.8.0.jar:lib/net.minidev.json-smart-2.4.10.jar:lib/net.minidev.accessors-smart-2.4.9.jar:lib/org.eclipse.jetty.jetty-util-9.4.53.v20231009.jar:lib/org.apache.kafka.kafka-clients-3.6.0.jar:lib/com.github.luben.zstd-jni-1.5.5-1.jar:lib/org.lz4.lz4-java-1.8.0.jar:lib/org.xerial.snappy.snappy-java-1.1.10.4.jar:lib/org.apache.zookeeper.zookeeper-3.8.3.jar:lib/org.apache.yetus.audience-annotations-0.12.0.jar:lib/commons-io.commons-io-2.11.0.jar:lib/org.apache.zookeeper.zookeeper-jute-3.8.3.jar:lib/io.netty.netty-transport-4.1.100.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.100.Final-linux-x86_64.jar:lib/io.netty.netty-transport-native-unix-common-4.1.100.Final.jar:lib/io.netty.netty-transport-classes-epoll-4.1.100.Final.jar:lib/io.micrometer.micrometer-core-1.9.5.jar:lib/org.hdrhistogram.HdrHistogram-2.1.12.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.micrometer.micrometer-registry-prometheus-1.9.5.jar:lib/io.prometheus.simpleclient_common-0.15.0.jar:lib/io.prometheus.simpleclient-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_common-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel_agent-0.15.0.jar:lib/io.vertx.vertx-micrometer-metrics-4.4.6.jar:lib/io.netty.netty-codec-4.1.100.Final.jar:lib/io.netty.netty-codec-socks-4.1.100.Final.jar io.strimzi.operator.cluster.Main
2024-02-14 10:38:05 INFO  Main:72 - ClusterOperator 0.38.0 is starting
2024-02-14 10:38:05 INFO  Util:224 - Using config:
        PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
        container: oci
        NEW_RELIC_METADATA_KUBERNETES_CONTAINER_IMAGE_NAME: quay.io/strimzi/operator:0.38.0
        NEW_RELIC_METADATA_KUBERNETES_NODE_NAME: aks-appnodepool-40837920-vmss00002p
        STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE: quay.io/strimzi/operator:0.38.0
        JAVA_OPTS:  -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom  -XX:+ExitOnOutOfMemoryError -XX:MinRAMPercentage=15 -XX:MaxRAMPercentage=20
        NEW_RELIC_METADATA_KUBERNETES_CLUSTER_NAME: aks-lab-infra
        NEW_RELIC_METADATA_KUBERNETES_DEPLOYMENT_NAME: strimzi-cluster-operator
        STRIMZI_HOME: /opt/strimzi
        PWD: /opt/strimzi
        STRIMZI_LEADER_ELECTION_IDENTITY: strimzi-cluster-operator-6b6864ddcf-w8njh
        KUBERNETES_PORT_443_TCP: tcp://10.0.0.1:443
        JAVA_MAIN: io.strimzi.operator.cluster.Main
        STRIMZI_LEADER_ELECTION_ENABLED: true
        STRIMZI_VERSION: 0.38.0
        STRIMZI_NAMESPACE: strimzi-kafka-1,strimzi-kafka-2,test,kafkaoperator
        STRIMZI_DEFAULT_KANIKO_EXECUTOR_IMAGE: quay.io/strimzi/kaniko-executor:0.38.0
        TINI_SHA256_S390X: 931b70a182af879ca249ae9de87ef68423121b38d235c78997fafc680ceab32d
        STRIMZI_DEFAULT_MAVEN_BUILDER: quay.io/strimzi/maven-builder:0.38.0
        STRIMZI_FEATURE_GATES:
        TINI_SHA256_AMD64: 93dcc18adc78c65a028a84799ecf8ad40c936fdfc5f2a57b1acda5a8117fa82c
        KUBERNETES_SERVICE_DNS_DOMAIN: cluster.local
        STRIMZI_LEADER_ELECTION_LEASE_NAME: strimzi-cluster-operator
        KUBERNETES_SERVICE_PORT_HTTPS: 443
        SHLVL: 0
        STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE: quay.io/strimzi/kafka-bridge:0.27.0
        KUBERNETES_PORT: tcp://10.0.0.1:443
        STRIMZI_KAFKA_MIRROR_MAKER_IMAGES: 3.5.0=quay.io/strimzi/kafka:0.38.0-kafka-3.5.0
3.5.1=quay.io/strimzi/kafka:0.38.0-kafka-3.5.1
3.6.0=quay.io/strimzi/kafka:0.38.0-kafka-3.6.0

        JAVA_HOME: /usr/lib/jvm/jre-17
        STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
        KUBERNETES_SERVICE_HOST: 10.0.0.1
        JAVA_CLASSPATH: :lib/io.strimzi.cluster-operator-0.38.0.jar:lib/io.strimzi.api-0.38.0.jar:lib/io.fabric8.kubernetes-model-apiextensions-6.9.0.jar:lib/io.strimzi.config-model-0.38.0.jar:lib/io.strimzi.operator-common-0.38.0.jar:lib/io.strimzi.crd-annotations-0.38.0.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar:lib/org.eclipse.jetty.jetty-server-9.4.53.v20231009.jar:lib/org.eclipse.jetty.jetty-http-9.4.53.v20231009.jar:lib/org.eclipse.jetty.jetty-io-9.4.53.v20231009.jar:lib/javax.servlet.javax.servlet-api-3.1.0.jar:lib/io.fabric8.openshift-client-6.9.0.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.fabric8.openshift-client-api-6.9.0.jar:lib/io.fabric8.openshift-model-clusterautoscaling-6.9.0.jar:lib/io.fabric8.openshift-model-operator-6.9.0.jar:lib/io.fabric8.openshift-model-operatorhub-6.9.0.jar:lib/io.fabric8.openshift-model-machine-6.9.0.jar:lib/io.fabric8.openshift-model-whereabouts-6.9.0.jar:lib/io.fabric8.openshift-model-monitoring-6.9.0.jar:lib/io.fabric8.openshift-model-storageversionmigrator-6.9.0.jar:lib/io.fabric8.openshift-model-tuned-6.9.0.jar:lib/io.fabric8.openshift-model-console-6.9.0.jar:lib/io.fabric8.openshift-model-config-6.9.0.jar:lib/io.fabric8.openshift-model-machineconfig-6.9.0.jar:lib/io.fabric8.openshift-model-miscellaneous-6.9.0.jar:lib/io.fabric8.openshift-model-hive-6.9.0.jar:lib/io.fabric8.openshift-model-installer-6.9.0.jar:lib/io.fabric8.kubernetes-client-6.9.0.jar:lib/io.fabric8.kubernetes-httpclient-jdk-6.9.0.jar:lib/io.fabric8.kubernetes-client-api-6.9.0.jar:lib/io.fabric8.kubernetes-model-gatewayapi-6.9.0.jar:lib/io.fabric8.kubernetes-model-resource-6.9.0.jar:lib/io.fabric8.kubernetes-model-admissionregistration-6.9.0.jar:lib/io.fabric8.kubernetes-model-autoscaling-6.9.0.jar:lib/io.fabric8.kubernetes-model-batch-6.9.0.jar:lib/io.fabric8.kubernetes-model-certificates-6.9.0.jar:lib/io.fabric8.kubernetes-model-discovery-6.9.0.jar:lib/io.fabric8.kubernetes-model-extensions-6.9.0.jar:lib/io.fabric8.kubernetes-model-flowcontrol-6.9.0.jar:lib/io.fabric8.kubernetes-model-metrics-6.9.0.jar:lib/io.fabric8.kubernetes-model-scheduling-6.9.0.jar:lib/io.fabric8.kubernetes-model-node-6.9.0.jar:lib/org.slf4j.slf4j-api-1.7.36.jar:lib/org.snakeyaml.snakeyaml-engine-2.7.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.15.2.jar:lib/io.fabric8.kubernetes-model-core-6.9.0.jar:lib/io.fabric8.kubernetes-model-common-6.9.0.jar:lib/io.fabric8.kubernetes-model-events-6.9.0.jar:lib/io.fabric8.kubernetes-model-policy-6.9.0.jar:lib/io.fabric8.kubernetes-model-rbac-6.9.0.jar:lib/io.fabric8.kubernetes-model-apps-6.9.0.jar:lib/io.fabric8.kubernetes-model-storageclass-6.9.0.jar:lib/io.fabric8.kubernetes-model-networking-6.9.0.jar:lib/io.fabric8.kubernetes-model-coordination-6.9.0.jar:lib/io.fabric8.openshift-model-6.9.0.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.15.2.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.15.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.15.2.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.15.2.jar:lib/org.yaml.snakeyaml-2.0.jar:lib/io.vertx.vertx-core-4.4.6.jar:lib/io.netty.netty-common-4.1.100.Final.jar:lib/io.netty.netty-buffer-4.1.100.Final.jar:lib/io.netty.netty-handler-4.1.100.Final.jar:lib/io.netty.netty-handler-proxy-4.1.100.Final.jar:lib/io.netty.netty-codec-http-4.1.100.Final.jar:lib/io.netty.netty-codec-http2-4.1.100.Final.jar:lib/io.netty.netty-resolver-4.1.100.Final.jar:lib/io.netty.netty-resolver-dns-4.1.100.Final.jar:lib/io.netty.netty-codec-dns-4.1.100.Final.jar:lib/io.strimzi.certificate-manager-0.38.0.jar:lib/org.apache.logging.log4j.log4j-core-2.17.2.jar:lib/org.apache.logging.log4j.log4j-api-2.17.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar:lib/io.strimzi.kafka-oauth-server-0.14.0.jar:lib/io.strimzi.kafka-oauth-server-plain-0.14.0.jar:lib/io.strimzi.kafka-oauth-client-0.14.0.jar:lib/io.strimzi.kafka-oauth-common-0.14.0.jar:lib/com.nimbusds.nimbus-jose-jwt-9.31.jar:lib/com.github.stephenc.jcip.jcip-annotations-1.0-1.jar:lib/com.jayway.jsonpath.json-path-2.8.0.jar:lib/net.minidev.json-smart-2.4.10.jar:lib/net.minidev.accessors-smart-2.4.9.jar:lib/org.eclipse.jetty.jetty-util-9.4.53.v20231009.jar:lib/org.apache.kafka.kafka-clients-3.6.0.jar:lib/com.github.luben.zstd-jni-1.5.5-1.jar:lib/org.lz4.lz4-java-1.8.0.jar:lib/org.xerial.snappy.snappy-java-1.1.10.4.jar:lib/org.apache.zookeeper.zookeeper-3.8.3.jar:lib/org.apache.yetus.audience-annotations-0.12.0.jar:lib/commons-io.commons-io-2.11.0.jar:lib/org.apache.zookeeper.zookeeper-jute-3.8.3.jar:lib/io.netty.netty-transport-4.1.100.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.100.Final-linux-x86_64.jar:lib/io.netty.netty-transport-native-unix-common-4.1.100.Final.jar:lib/io.netty.netty-transport-classes-epoll-4.1.100.Final.jar:lib/io.micrometer.micrometer-core-1.9.5.jar:lib/org.hdrhistogram.HdrHistogram-2.1.12.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.micrometer.micrometer-registry-prometheus-1.9.5.jar:lib/io.prometheus.simpleclient_common-0.15.0.jar:lib/io.prometheus.simpleclient-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_common-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel_agent-0.15.0.jar:lib/io.vertx.vertx-micrometer-metrics-4.4.6.jar:lib/io.netty.netty-codec-4.1.100.Final.jar:lib/io.netty.netty-codec-socks-4.1.100.Final.jar
        NEW_RELIC_METADATA_KUBERNETES_CONTAINER_NAME: strimzi-cluster-operator
        STRIMZI_DEFAULT_CRUISE_CONTROL_IMAGE: vel123/cruise-control-ui:latest
        TINI_VERSION: v0.19.0
        STRIMZI_OPERATION_TIMEOUT_MS: 300000
        KUBERNETES_PORT_443_TCP_ADDR: 10.0.0.1
        NEW_RELIC_METADATA_KUBERNETES_POD_NAME: strimzi-cluster-operator-6b6864ddcf-w8njh
        STRIMZI_LEADER_ELECTION_LEASE_NAMESPACE: kafkaoperator
        STRIMZI_KAFKA_MIRROR_MAKER_2_IMAGES: 3.5.0=quay.io/strimzi/kafka:0.38.0-kafka-3.5.0
3.5.1=quay.io/strimzi/kafka:0.38.0-kafka-3.5.1
3.6.0=quay.io/strimzi/kafka:0.38.0-kafka-3.6.0

        KUBERNETES_PORT_443_TCP_PROTO: tcp
        STRIMZI_DEFAULT_USER_OPERATOR_IMAGE: quay.io/strimzi/operator:0.38.0
        KUBERNETES_SERVICE_PORT: 443
        STRIMZI_DEFAULT_KAFKA_EXPORTER_IMAGE: quay.io/strimzi/kafka:0.38.0-kafka-3.6.0
        TINI_SHA256_ARM64: 07952557df20bfd2a95f9bef198b445e006171969499a1d361bd9e6f8e5e0e81
        NEW_RELIC_METADATA_KUBERNETES_NAMESPACE_NAME: kafkaoperator
        STRIMZI_DEFAULT_KAFKA_INIT_IMAGE: quay.io/strimzi/operator:0.38.0
        TINI_SHA256_PPC64LE: 3f658420974768e40810001a038c29d003728c5fe86da211cff5059e48cfdfde
        HOSTNAME: strimzi-cluster-operator-6b6864ddcf-w8njh
        STRIMZI_KAFKA_CONNECT_IMAGES: 3.5.0=quay.io/strimzi/kafka:0.38.0-kafka-3.5.0
3.5.1=quay.io/strimzi/kafka:0.38.0-kafka-3.5.1
3.6.0=quay.io/strimzi/kafka:0.38.0-kafka-3.6.0

        STRIMZI_KAFKA_IMAGES: 3.5.0=quay.io/strimzi/kafka:0.38.0-kafka-3.5.0
3.5.1=quay.io/strimzi/kafka:0.38.0-kafka-3.5.1
3.6.0=quay.io/strimzi/kafka:0.38.0-kafka-3.6.0

        KUBERNETES_PORT_443_TCP_PORT: 443
        STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE: quay.io/strimzi/kafka:0.38.0-kafka-3.6.0
        STRIMZI_OPERATOR_NAME: strimzi-cluster-operator-6b6864ddcf-w8njh
        HOME: /home/strimzi
        MALLOC_ARENA_MAX: 2
        STRIMZI_OPERATOR_NAMESPACE: kafkaoperator

2024-02-14 10:38:05 INFO  Main:75 - Cluster Operator configuration is ClusterOperatorConfig{
        namespaces='[test, kafkaoperator, strimzi-kafka-1, strimzi-kafka-2]'
        reconciliationIntervalMs=120000
        operationTimeoutMs=300000
        connectBuildTimeoutMs=300000
        createClusterRoles=false
        networkPolicyGeneration=true
        versions='versions{3.5.0={proto: 3.5 msg: 3.5 kafka-image: quay.io/strimzi/kafka:0.38.0-kafka-3.5.0 connect-image: quay.io/strimzi/kafka:0.38.0-kafka-3.5.0 mirrormaker-image: quay.io/strimzi/kafka:0.38.0-kafka-3.5.0 mirrormaker2-image: quay.io/strimzi/kafka:0.38.0-kafka-3.5.0}, 3.5.1={proto: 3.5 msg: 3.5 kafka-image: quay.io/strimzi/kafka:0.38.0-kafka-3.5.1 connect-image: quay.io/strimzi/kafka:0.38.0-kafka-3.5.1 mirrormaker-image: quay.io/strimzi/kafka:0.38.0-kafka-3.5.1 mirrormaker2-image: quay.io/strimzi/kafka:0.38.0-kafka-3.5.1}, 3.6.0={proto: 3.6 msg: 3.6 kafka-image: quay.io/strimzi/kafka:0.38.0-kafka-3.6.0 connect-image: quay.io/strimzi/kafka:0.38.0-kafka-3.6.0 mirrormaker-image: quay.io/strimzi/kafka:0.38.0-kafka-3.6.0 mirrormaker2-image: quay.io/strimzi/kafka:0.38.0-kafka-3.6.0}}'
        imagePullPolicy='null'
        imagePullSecrets='null'
        operatorNamespace='kafkaoperator'
        operatorNamespaceLabels='null'
        customResourceSelector='null'
        featureGates='FeatureGates(UseKRaft=false,StableConnectIdentities=true,KafkaNodePools=false,UnidirectionalTopicOperator=false)'
        zkAdminSessionTimeoutMs=10000
        dnsCacheTtlSec=30
        podSetReconciliationOnly=false
        podSetControllerWorkQueueSize=1024
        operatorName='strimzi-cluster-operator-6b6864ddcf-w8njh'
        podSecurityProviderClass='io.strimzi.plugin.security.profiles.impl.BaselinePodSecurityProvider'
        leaderElectionConfig='LeaderElectionConfig{leaseName='strimzi-cluster-operator', namespace='kafkaoperator', identity='strimzi-cluster-operator-6b6864ddcf-w8njh', leaseDuration=PT15S, renewDeadline=PT10S, retryPeriod=PT2S}'}
2024-02-14 10:38:07 INFO  Main:345 - Health and metrics server is ready on port 8080)
2024-02-14 10:38:07 INFO  Main:255 - Waiting to become a leader
2024-02-14 10:38:07 INFO  LeaderElectionManager:69 - Starting the Leader Elector
2024-02-14 10:38:08 INFO  LeaderElectionManager:121 - The new leader is strimzi-cluster-operator-5f8d459678-6h5rs
2024-02-14 10:38:30 INFO  LeaderElectionManager:121 - The new leader is
2024-02-14 10:38:30 INFO  LeaderElectionManager:121 - The new leader is strimzi-cluster-operator-6b6864ddcf-w8njh
2024-02-14 10:38:30 INFO  LeaderElectionManager:103 - Started being a leader
2024-02-14 10:38:30 INFO  Main:237 - I'm the new leader
2024-02-14 10:38:30 WARN  PlatformFeaturesAvailability:164 - API Group route.openshift.io is not supported
2024-02-14 10:38:30 WARN  PlatformFeaturesAvailability:164 - API Group build.openshift.io is not supported
2024-02-14 10:38:30 WARN  PlatformFeaturesAvailability:164 - API Group image.openshift.io is not supported
2024-02-14 10:38:30 INFO  Main:124 - Environment facts gathered: PlatformFeaturesAvailability(KubernetesVersion=1.26,OpenShiftRoutes=false,OpenShiftBuilds=false,OpenShiftImageStreams=false)
2024-02-14 10:38:30 INFO  PodSecurityProviderFactory:43 - Found PodSecurityProvider io.strimzi.plugin.security.profiles.impl.BaselinePodSecurityProvider
2024-02-14 10:38:30 INFO  PodSecurityProviderFactory:62 - Initializing PodSecurityProvider io.strimzi.plugin.security.profiles.impl.BaselinePodSecurityProvider
2024-02-14 10:38:30 INFO  ClusterOperator:86 - Creating ClusterOperator for namespace test
2024-02-14 10:38:30 INFO  ClusterOperator:100 - Starting ClusterOperator for namespace test
2024-02-14 10:38:30 INFO  ClusterOperator:86 - Creating ClusterOperator for namespace kafkaoperator
2024-02-14 10:38:30 INFO  ClusterOperator:86 - Creating ClusterOperator for namespace strimzi-kafka-1
2024-02-14 10:38:30 INFO  ClusterOperator:86 - Creating ClusterOperator for namespace strimzi-kafka-2
2024-02-14 10:38:30 INFO  ClusterOperator:100 - Starting ClusterOperator for namespace kafkaoperator
2024-02-14 10:38:30 INFO  ClusterOperator:100 - Starting ClusterOperator for namespace strimzi-kafka-1
2024-02-14 10:38:30 INFO  ClusterOperator:100 - Starting ClusterOperator for namespace strimzi-kafka-2
2024-02-14 10:38:30 INFO  StrimziPodSetController:583 - Starting the StrimziPodSet controller
2024-02-14 10:38:30 INFO  StrimziPodSetController:583 - Starting the StrimziPodSet controller
2024-02-14 10:38:30 INFO  StrimziPodSetController:583 - Starting the StrimziPodSet controller
2024-02-14 10:38:30 INFO  StrimziPodSetController:583 - Starting the StrimziPodSet controller
2024-02-14 10:38:30 INFO  StrimziPodSetController:552 - Starting StrimziPodSet controller for namespace strimzi-kafka-2
2024-02-14 10:38:30 INFO  StrimziPodSetController:552 - Starting StrimziPodSet controller for namespace strimzi-kafka-1
2024-02-14 10:38:30 WARN  VersionUsageUtils:60 - The client is using resource type 'strimzipodsets' with unstable version 'v1beta2'
2024-02-14 10:38:30 INFO  StrimziPodSetController:552 - Starting StrimziPodSet controller for namespace kafkaoperator
2024-02-14 10:38:30 INFO  StrimziPodSetController:552 - Starting StrimziPodSet controller for namespace test
2024-02-14 10:38:30 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkas' with unstable version 'v1beta2'
2024-02-14 10:38:30 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkaconnects' with unstable version 'v1beta2'
2024-02-14 10:38:30 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkamirrormaker2s' with unstable version 'v1beta2'
2024-02-14 10:38:30 INFO  StrimziPodSetController:555 - Waiting for informers to sync
2024-02-14 10:38:30 INFO  StrimziPodSetController:555 - Waiting for informers to sync
2024-02-14 10:38:30 INFO  StrimziPodSetController:555 - Waiting for informers to sync
2024-02-14 10:38:30 INFO  StrimziPodSetController:555 - Waiting for informers to sync
2024-02-14 10:38:31 INFO  ClusterOperator:114 - Opened watch for Kafka operator
2024-02-14 10:38:31 INFO  ClusterOperator:114 - Opened watch for Kafka operator
2024-02-14 10:38:31 INFO  ClusterOperator:114 - Opened watch for KafkaMirrorMaker operator
2024-02-14 10:38:31 INFO  ClusterOperator:114 - Opened watch for KafkaMirrorMaker operator
2024-02-14 10:38:31 INFO  ClusterOperator:114 - Opened watch for KafkaConnect operator
2024-02-14 10:38:31 INFO  ClusterOperator:114 - Opened watch for KafkaConnect operator
2024-02-14 10:38:31 INFO  ClusterOperator:114 - Opened watch for KafkaBridge operator
2024-02-14 10:38:31 INFO  ClusterOperator:114 - Opened watch for KafkaBridge operator
2024-02-14 10:38:31 INFO  ClusterOperator:114 - Opened watch for KafkaMirrorMaker2 operator
2024-02-14 10:38:31 INFO  ClusterOperator:114 - Opened watch for KafkaMirrorMaker2 operator
2024-02-14 10:38:31 INFO  ClusterOperator:114 - Opened watch for KafkaRebalance operator
2024-02-14 10:38:31 INFO  ClusterOperator:114 - Opened watch for KafkaRebalance operator
2024-02-14 10:38:32 INFO  ClusterOperator:131 - Opened watch for KafkaConnector operator
2024-02-14 10:38:32 INFO  ClusterOperator:139 - Setting up periodic reconciliation for namespace kafkaoperator
2024-02-14 10:38:32 INFO  Main:206 - Cluster Operator verticle started in namespace kafkaoperator without label selector
2024-02-14 10:38:32 INFO  ClusterOperator:131 - Opened watch for KafkaConnector operator
2024-02-14 10:38:32 INFO  ClusterOperator:139 - Setting up periodic reconciliation for namespace test
2024-02-14 10:38:32 INFO  Main:206 - Cluster Operator verticle started in namespace test without label selector
2024-02-14 10:38:37 INFO  ClusterOperator:114 - Opened watch for Kafka operator
2024-02-14 10:38:37 INFO  AbstractOperator:511 - Reconciliation #1(watch) Kafka(strimzi-kafka-2/poc-strimizi-2): Kafka poc-strimizi-2 in namespace strimzi-kafka-2 was ADDED
2024-02-14 10:38:37 INFO  AbstractOperator:511 - Reconciliation #2(watch) Kafka(strimzi-kafka-1/poc-strimizi): Kafka poc-strimizi in namespace strimzi-kafka-1 was ADDED
2024-02-14 10:38:37 INFO  ClusterOperator:114 - Opened watch for Kafka operator
2024-02-14 10:38:37 INFO  AbstractOperator:511 - Reconciliation #3(watch) Kafka(test/test-strimzi): Kafka test-strimzi in namespace test was ADDED
2024-02-14 10:38:37 INFO  ClusterOperator:114 - Opened watch for KafkaMirrorMaker operator
2024-02-14 10:38:37 INFO  ClusterOperator:114 - Opened watch for KafkaMirrorMaker operator
2024-02-14 10:38:37 INFO  ClusterOperator:114 - Opened watch for KafkaConnect operator
2024-02-14 10:38:37 INFO  ClusterOperator:114 - Opened watch for KafkaConnect operator
2024-02-14 10:38:37 INFO  StrimziPodSetController:560 - Informers are in-sync
2024-02-14 10:38:37 INFO  StrimziPodSetController:560 - Informers are in-sync
2024-02-14 10:38:37 INFO  ClusterOperator:114 - Opened watch for KafkaBridge operator
2024-02-14 10:38:37 INFO  ClusterOperator:114 - Opened watch for KafkaBridge operator
2024-02-14 10:38:37 INFO  StrimziPodSetController:378 - Reconciliation #4(watch) StrimziPodSet(strimzi-kafka-1/poc-strimizi-kafka): StrimziPodSet will be reconciled
2024-02-14 10:38:38 INFO  ClusterOperator:114 - Opened watch for KafkaMirrorMaker2 operator
2024-02-14 10:38:38 INFO  ClusterOperator:114 - Opened watch for KafkaMirrorMaker2 operator
2024-02-14 10:38:38 INFO  StrimziPodSetController:560 - Informers are in-sync
2024-02-14 10:38:38 INFO  StrimziPodSetController:378 - Reconciliation #5(watch) StrimziPodSet(strimzi-kafka-2/poc-strimizi-2-kafka): StrimziPodSet will be reconciled
2024-02-14 10:38:38 INFO  ClusterOperator:114 - Opened watch for KafkaRebalance operator
2024-02-14 10:38:38 INFO  ClusterOperator:114 - Opened watch for KafkaRebalance operator
2024-02-14 10:38:38 INFO  StrimziPodSetController:560 - Informers are in-sync
2024-02-14 10:38:38 INFO  StrimziPodSetController:378 - Reconciliation #6(watch) StrimziPodSet(test/test-strimzi-zookeeper): StrimziPodSet will be reconciled
2024-02-14 10:38:38 INFO  ClusterOperator:131 - Opened watch for KafkaConnector operator
2024-02-14 10:38:38 INFO  ClusterOperator:139 - Setting up periodic reconciliation for namespace strimzi-kafka-2
2024-02-14 10:38:38 INFO  Main:206 - Cluster Operator verticle started in namespace strimzi-kafka-2 without label selector
2024-02-14 10:38:38 INFO  ClusterOperator:131 - Opened watch for KafkaConnector operator
2024-02-14 10:38:38 INFO  ClusterOperator:139 - Setting up periodic reconciliation for namespace strimzi-kafka-1
2024-02-14 10:38:38 INFO  Main:206 - Cluster Operator verticle started in namespace strimzi-kafka-1 without label selector
2024-02-14 10:38:38 INFO  StrimziPodSetController:414 - Reconciliation #4(watch) StrimziPodSet(strimzi-kafka-1/poc-strimizi-kafka): reconciled
2024-02-14 10:38:38 INFO  StrimziPodSetController:414 - Reconciliation #6(watch) StrimziPodSet(test/test-strimzi-zookeeper): reconciled
2024-02-14 10:38:38 INFO  StrimziPodSetController:414 - Reconciliation #5(watch) StrimziPodSet(strimzi-kafka-2/poc-strimizi-2-kafka): reconciled
2024-02-14 10:38:38 INFO  AbstractOperator:265 - Reconciliation #3(watch) Kafka(test/test-strimzi): Kafka test-strimzi will be checked for creation or modification
2024-02-14 10:38:38 INFO  AbstractOperator:265 - Reconciliation #2(watch) Kafka(strimzi-kafka-1/poc-strimizi): Kafka poc-strimizi will be checked for creation or modification
2024-02-14 10:38:38 INFO  StrimziPodSetController:378 - Reconciliation #7(watch) StrimziPodSet(test/test-strimzi-kafka): StrimziPodSet will be reconciled
2024-02-14 10:38:38 INFO  StrimziPodSetController:414 - Reconciliation #7(watch) StrimziPodSet(test/test-strimzi-kafka): reconciled
2024-02-14 10:38:38 INFO  StrimziPodSetController:378 - Reconciliation #8(watch) StrimziPodSet(strimzi-kafka-2/poc-strimizi-2-zookeeper): StrimziPodSet will be reconciled
2024-02-14 10:38:38 INFO  StrimziPodSetController:414 - Reconciliation #8(watch) StrimziPodSet(strimzi-kafka-2/poc-strimizi-2-zookeeper): reconciled
2024-02-14 10:38:38 INFO  AbstractOperator:265 - Reconciliation #1(watch) Kafka(strimzi-kafka-2/poc-strimizi-2): Kafka poc-strimizi-2 will be checked for creation or modification
2024-02-14 10:38:42 INFO  StrimziPodSetController:378 - Reconciliation #9(watch) StrimziPodSet(strimzi-kafka-1/poc-strimizi-kafka): StrimziPodSet will be reconciled
2024-02-14 10:38:42 INFO  StrimziPodSetController:414 - Reconciliation #9(watch) StrimziPodSet(strimzi-kafka-1/poc-strimizi-kafka): reconciled
2024-02-14 10:38:45 INFO  KafkaRoller:535 - Reconciliation #2(watch) Kafka(strimzi-kafka-1/poc-strimizi): Pod poc-strimizi-kafka-3/3 needs to be restarted, because it seems to be stuck and restart might help
2024-02-14 10:38:45 INFO  KafkaRoller:339 - Reconciliation #2(watch) Kafka(strimzi-kafka-1/poc-strimizi): Could not verify pod poc-strimizi-kafka-3/3 is up-to-date, giving up after 10 attempts. Total delay between attempts 127750ms
io.strimzi.operator.cluster.operator.resource.KafkaRoller$FatalProblem: Pod is unschedulable or is not starting
        at io.strimzi.operator.cluster.operator.resource.KafkaRoller.checkReconfigurability(KafkaRoller.java:543) ~[io.strimzi.cluster-operator-0.38.0.jar:0.38.0]
        at io.strimzi.operator.cluster.operator.resource.KafkaRoller.restartIfNecessary(KafkaRoller.java:418) ~[io.strimzi.cluster-operator-0.38.0.jar:0.38.0]
        at io.strimzi.operator.cluster.operator.resource.KafkaRoller.lambda$schedule$6(KafkaRoller.java:333) ~[io.strimzi.cluster-operator-0.38.0.jar:0.38.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-02-14 10:38:45 ERROR AbstractOperator:284 - Reconciliation #2(watch) Kafka(strimzi-kafka-1/poc-strimizi): createOrUpdate failed
io.strimzi.operator.cluster.operator.resource.KafkaRoller$FatalProblem: Pod is unschedulable or is not starting
        at io.strimzi.operator.cluster.operator.resource.KafkaRoller.checkReconfigurability(KafkaRoller.java:543) ~[io.strimzi.cluster-operator-0.38.0.jar:0.38.0]
        at io.strimzi.operator.cluster.operator.resource.KafkaRoller.restartIfNecessary(KafkaRoller.java:418) ~[io.strimzi.cluster-operator-0.38.0.jar:0.38.0]
        at io.strimzi.operator.cluster.operator.resource.KafkaRoller.lambda$schedule$6(KafkaRoller.java:333) ~[io.strimzi.cluster-operator-0.38.0.jar:0.38.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-02-14 10:38:46 WARN  AbstractOperator:557 - Reconciliation #2(watch) Kafka(strimzi-kafka-1/poc-strimizi): Failed to reconcile
io.strimzi.operator.cluster.operator.resource.KafkaRoller$FatalProblem: Pod is unschedulable or is not starting
        at io.strimzi.operator.cluster.operator.resource.KafkaRoller.checkReconfigurability(KafkaRoller.java:543) ~[io.strimzi.cluster-operator-0.38.0.jar:0.38.0]
        at io.strimzi.operator.cluster.operator.resource.KafkaRoller.restartIfNecessary(KafkaRoller.java:418) ~[io.strimzi.cluster-operator-0.38.0.jar:0.38.0]
        at io.strimzi.operator.cluster.operator.resource.KafkaRoller.lambda$schedule$6(KafkaRoller.java:333) ~[io.strimzi.cluster-operator-0.38.0.jar:0.38.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-02-14 10:38:48 INFO  KafkaRoller:355 - Reconciliation #1(watch) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-0/0 is up-to-date due to ForceableProblem: Pod poc-strimizi-2-kafka-0 is controller and there are other pods to verify. Non-controller pods will be verified first, retrying after at least 250ms
2024-02-14 10:38:49 WARN  AbstractModel:286 - Reconciliation #3(watch) Kafka(test/test-strimzi): Anomaly goals contained goals which are not in the configured default goals. Anomaly goals have been changed to match the specified default goals.
2024-02-14 10:38:49 INFO  KafkaAvailability:135 - Reconciliation #1(watch) Kafka(strimzi-kafka-2/poc-strimizi-2): mm2-offset-syncs.poc-strimzi-1.internal/0 will be under-replicated (ISR={1}, replicas=[1,0,2], min.insync.replicas=1) if broker 1 is restarted.
2024-02-14 10:38:49 INFO  KafkaRoller:355 - Reconciliation #1(watch) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-1/1 is up-to-date due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod poc-strimizi-2-kafka-1 cannot be updated right now., retrying after at least 250ms
2024-02-14 10:38:50 INFO  AbstractOperator:537 - Reconciliation #3(watch) Kafka(test/test-strimzi): reconciled
2024-02-14 10:38:50 INFO  StrimziPodSetController:378 - Reconciliation #10(watch) StrimziPodSet(strimzi-kafka-1/poc-strimizi-kafka): StrimziPodSet will be reconciled
2024-02-14 10:38:50 INFO  StrimziPodSetController:414 - Reconciliation #10(watch) StrimziPodSet(strimzi-kafka-1/poc-strimizi-kafka): reconciled
2024-02-14 10:38:51 INFO  KafkaRoller:355 - Reconciliation #1(watch) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-0/0 is up-to-date due to ForceableProblem: Pod poc-strimizi-2-kafka-0 is controller and there are other pods to verify. Non-controller pods will be verified first, retrying after at least 500ms
2024-02-14 10:38:52 INFO  KafkaAvailability:135 - Reconciliation #1(watch) Kafka(strimzi-kafka-2/poc-strimizi-2): mm2-offset-syncs.poc-strimzi-1.internal/0 will be under-replicated (ISR={1}, replicas=[1,0,2], min.insync.replicas=1) if broker 1 is restarted.
2024-02-14 10:38:52 INFO  KafkaRoller:355 - Reconciliation #1(watch) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-1/1 is up-to-date due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod poc-strimizi-2-kafka-1 cannot be updated right now., retrying after at least 500ms
2024-02-14 10:38:53 INFO  KafkaRoller:355 - Reconciliation #1(watch) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-0/0 is up-to-date due to ForceableProblem: Pod poc-strimizi-2-kafka-0 is controller and there are other pods to verify. Non-controller pods will be verified first, retrying after at least 1000ms
2024-02-14 10:38:54 INFO  KafkaAvailability:135 - Reconciliation #1(watch) Kafka(strimzi-kafka-2/poc-strimizi-2): mm2-offset-syncs.poc-strimzi-1.internal/0 will be under-replicated (ISR={1}, replicas=[1,0,2], min.insync.replicas=1) if broker 1 is restarted.
2024-02-14 10:38:54 INFO  KafkaRoller:355 - Reconciliation #1(watch) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-1/1 is up-to-date due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod poc-strimizi-2-kafka-1 cannot be updated right now., retrying after at least 1000ms
2024-02-14 10:38:54 INFO  KafkaRoller:355 - Reconciliation #1(watch) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-0/0 is up-to-date due to ForceableProblem: Pod poc-strimizi-2-kafka-0 is controller and there are other pods to verify. Non-controller pods will be verified first, retrying after at least 2000ms
2024-02-14 10:38:56 INFO  KafkaAvailability:135 - Reconciliation #1(watch) Kafka(strimzi-kafka-2/poc-strimizi-2): mm2-offset-syncs.poc-strimzi-1.internal/0 will be under-replicated (ISR={1}, replicas=[1,0,2], min.insync.replicas=1) if broker 1 is restarted.
2024-02-14 10:38:56 INFO  KafkaRoller:355 - Reconciliation #1(watch) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-1/1 is up-to-date due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod poc-strimizi-2-kafka-1 cannot be updated right now., retrying after at least 2000ms
2024-02-14 10:38:57 INFO  KafkaRoller:355 - Reconciliation #1(watch) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-0/0 is up-to-date due to ForceableProblem: Pod poc-strimizi-2-kafka-0 is controller and there are other pods to verify. Non-controller pods will be verified first, retrying after at least 4000ms
2024-02-14 10:38:59 INFO  KafkaAvailability:135 - Reconciliation #1(watch) Kafka(strimzi-kafka-2/poc-strimizi-2): mm2-offset-syncs.poc-strimzi-1.internal/0 will be under-replicated (ISR={1}, replicas=[1,0,2], min.insync.replicas=1) if broker 1 is restarted.
2024-02-14 10:38:59 INFO  KafkaRoller:355 - Reconciliation #1(watch) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-1/1 is up-to-date due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod poc-strimizi-2-kafka-1 cannot be updated right now., retrying after at least 4000ms
2024-02-14 10:39:02 INFO  KafkaRoller:355 - Reconciliation #1(watch) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-0/0 is up-to-date due to ForceableProblem: Pod poc-strimizi-2-kafka-0 is controller and there are other pods to verify. Non-controller pods will be verified first, retrying after at least 8000ms
2024-02-14 10:39:03 INFO  KafkaAvailability:135 - Reconciliation #1(watch) Kafka(strimzi-kafka-2/poc-strimizi-2): mm2-offset-syncs.poc-strimzi-1.internal/0 will be under-replicated (ISR={1}, replicas=[1,0,2], min.insync.replicas=1) if broker 1 is restarted.
2024-02-14 10:39:03 INFO  KafkaRoller:355 - Reconciliation #1(watch) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-1/1 is up-to-date due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod poc-strimizi-2-kafka-1 cannot be updated right now., retrying after at least 8000ms
2024-02-14 10:39:10 INFO  StrimziPodSetController:378 - Reconciliation #11(watch) StrimziPodSet(strimzi-kafka-1/poc-strimizi-kafka): StrimziPodSet will be reconciled
2024-02-14 10:39:10 INFO  StrimziPodSetController:414 - Reconciliation #11(watch) StrimziPodSet(strimzi-kafka-1/poc-strimizi-kafka): reconciled
2024-02-14 10:39:10 INFO  KafkaRoller:355 - Reconciliation #1(watch) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-0/0 is up-to-date due to ForceableProblem: Pod poc-strimizi-2-kafka-0 is controller and there are other pods to verify. Non-controller pods will be verified first, retrying after at least 16000ms
2024-02-14 10:39:12 INFO  KafkaAvailability:135 - Reconciliation #1(watch) Kafka(strimzi-kafka-2/poc-strimizi-2): mm2-offset-syncs.poc-strimzi-1.internal/0 will be under-replicated (ISR={1}, replicas=[1,0,2], min.insync.replicas=1) if broker 1 is restarted.
2024-02-14 10:39:12 INFO  KafkaRoller:355 - Reconciliation #1(watch) Kafka(strimzi-kafka-2/poc-strimizi-2): Will temporarily skip verifying pod poc-strimizi-2-kafka-1/1 is up-to-date due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod poc-strimizi-2-kafka-1 cannot be updated right now., retrying after at least 16000ms

1 reply

scholzj Feb 14, 2024
Maintainer

Well, the broker log didn't even start Kafka. So if that is a complete log from when it terminated, I would say lack of resources might be an issue. You should probably configure resources for the Kafka cluster and all of its components in the Kafka CR.

Veljen · 2024-02-14T11:15:10Z

Veljen
Feb 14, 2024
Author

I have not set any resource but you are right, there is some issue with one of the worker node. There was some runtime problem and its not properly start the pod. I removed this node and now I am able to spin up the new broker. I was not able to see any errros in the kafka or resource or pod so I couldnt trace it. Thanks

  ----     ------                  ----                  ----                                     -------
  Normal   KubeletIsUp             23m (x15 over 6h43m)  kubelet-custom-plugin-monitor            Node condition KubeletProblem is now: Unknown, reason: KubeletIsUp, message: "Timeout when running plugin \"/etc/node-problem-detector.d/plugin/check_kubelet.s"
  Normal   KubeletIsUp             22m (x15 over 6h42m)  kubelet-custom-plugin-monitor            Node condition KubeletProblem is now: False, reason: KubeletIsUp, message: "kubelet service is up"
  Normal   ContainerRuntimeIsUp    22m (x21 over 6h43m)  container-runtime-custom-plugin-monitor  Node condition ContainerRuntimeProblem is now: Unknown, reason: ContainerRuntimeIsUp, message: "Timeout when running plugin \"/etc/node-problem-detector.d/plugin/check_runtime.s"
  Normal   ContainerRuntimeIsUp    21m (x21 over 6h42m)  container-runtime-custom-plugin-monitor  Node condition ContainerRuntimeProblem is now: False, reason: ContainerRuntimeIsUp, message: "container runtime service is up"
  Warning  ContainerRuntimeIsDown  14m (x23 over 6h43m)  container-runtime-custom-plugin-monitor  Timeout when running plugin "/etc/node-problem-detector.d/plugin/check_runtime.s
  Warning  KubeletIsDown           13m (x22 over 6h43m)  kubelet-custom-plugin-monitor            Timeout when running plugin "/etc/node-problem-detector.d/plugin/check_kubelet.s
  Warning  PodIPProblem            12m (x33 over 6h42m)  podip-problem-monitor                    Timeout when running plugin "/etc/node-problem-detector.d/plugin/check_pod_ips.sh": state - signal: killed. output - ""

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

adding 4th broker crashing in 0.38 operator #9685

{{title}}

Replies: 3 comments 1 reply

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Strimzi

adding 4th broker crashing in 0.38 operator #9685

Veljen Feb 14, 2024

Replies: 3 comments · 1 reply

scholzj Feb 14, 2024 Maintainer

Veljen Feb 14, 2024 Author

scholzj Feb 14, 2024 Maintainer

Veljen Feb 14, 2024 Author

Veljen
Feb 14, 2024

Replies: 3 comments 1 reply

scholzj
Feb 14, 2024
Maintainer

Veljen
Feb 14, 2024
Author

scholzj Feb 14, 2024
Maintainer

Veljen
Feb 14, 2024
Author