[Bug]: Kafka Pods fails to restart to fetch latest configuration

[sachintandon-nexla]

Bug Description

Strimzi operator fails to restart kafka strimzipodsets automatically after the kafka configuration is updated . fails with below error.

Steps to reproduce

1. updated the configuration in helm charts repo
2. argoCd picks up the change for the new version of kafka
3. kafka zk pods get restarted with the new configuration
4. kafka pods fails to restart with the new configuration all broker fails with the same error.

Expected behavior

No response

Strimzi version

0.39

Kubernetes version

1.28

Installation method

argoCD Helm Charts

Infrastructure

Amazon EKS

Configuration files and logs

No response

Additional context

Strimzi configuration :

strimzi-kafka-operator:
  enabled: true
  defaultImageRegistry: quay.io
  defaultImageRepository: strimzi
  defaultImageTag: 0.39.0
  featureGates: "+UnidirectionalTopicOperator"
  fullReconciliationIntervalMs: "3600000"
  operationTimeoutMs: "1800000"
  dashboards:
    enabled: true
    namespace: monitoring
    label: grafana_dashboard
    labelValue: "1"
  image:
    name: operator
    repository: ""
    #imagePullSecrets:
    # - name: secretname
  resources:
    limits:
      memory: 6Gi
      cpu: 3
    requests:
      memory: 5Gi
      cpu: 1
  watchNamespaces:
    - kube-system
  extraEnvs:
    - name: JAVA_OPTS
      value: "-Xms3g -Xmx3g"
    - name: STRIMZI_ZOOKEEPER_SESSION_TIMEOUT_MS
      value: "900000"
    - name: KUBERNETES_CONNECTION_TIMEOUT
      value: "90000"
    - name: KUBERNETES_WEBSOCKET_TIMEOUT
      value: "50000"
    - name: KUBERNETES_REQUEST_TIMEOUT
      value: "90000"
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
            - key: eks.amazonaws.com/nodegroup
              operator: In
              values:
                - Kafka
  tolerations:
  - key: "kafka"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"
  - key: "kafka"
    operator: "Equal"
    value: "true"
    effect: "NoExecute"
namespace: kube-system
clusterName: kafka-cluster

Kafka Strimzi : 0.39
Kafka Version : 3.6.1
Error Logs from strimzi:

2024-02-01 23:46:43 INFO  AbstractOperator:400 - Reconciliation #2(watch) Kafka(nexla/kafka-cluster): Reconciliation is in progress
2024-02-01 23:46:48 WARN  KafkaAvailability:90 - Reconciliation #2(watch) Kafka(nexla/kafka-cluster): Error determining whether it is safe to restart pod 3
java.util.concurrent.CompletionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=describeConfigs, deadlineMs=1706831208142, tries=2, nextAllowedTryMs=1706831208259) timed out at 1706831208159 after 2 attempt(s)
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
	at org.apache.kafka.common.internals.KafkaCompletableFuture.kafkaCompleteExceptionally(KafkaCompletableFuture.java:49) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl.completeExceptionally(KafkaFutureImpl.java:130) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.common.KafkaFuture.lambda$allOf$2(KafkaFuture.java:93) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
	at org.apache.kafka.common.internals.KafkaCompletableFuture.kafkaCompleteExceptionally(KafkaCompletableFuture.java:49) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl.completeExceptionally(KafkaFutureImpl.java:130) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient.lambda$completeAllExceptionally$1(KafkaAdminClient.java:406) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1779) ~[?:?]
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762) ~[?:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient.completeAllExceptionally(KafkaAdminClient.java:406) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient.completeAllExceptionally(KafkaAdminClient.java:395) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient.access$3000(KafkaAdminClient.java:284) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$11.handleFailure(KafkaAdminClient.java:2380) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$Call.handleTimeoutFailure(KafkaAdminClient.java:821) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$Call.fail(KafkaAdminClient.java:785) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.handleResponses(KafkaAdminClient.java:1230) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1389) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1312) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=describeConfigs, deadlineMs=1706831208142, tries=2, nextAllowedTryMs=1706831208259) timed out at 1706831208159 after 2 attempt(s)
Caused by: org.apache.kafka.common.errors.DisconnectException: Cancelled describeConfigs request with correlation id 66 due to node 1 being disconnected
2024-02-01 23:46:48 INFO  KafkaRoller:396 - Reconciliation #2(watch) Kafka(nexla/kafka-cluster): Will temporarily skip verifying pod kafka-cluster-kafka-3/3 is up-to-date due to ForceableProblem: An error while trying to determine the possibility of updating Kafka broker pods, retrying after at least 500ms

[scholzj]

You should probably share the full log and the actual Strimzi configuration => the Kafka CR. The error suggests the broker is not responding.

[sachintandon-nexla]

Full logs:

Auto-detected KUBERNETES_SERVICE_DNS_DOMAIN: cluster.local
+ exec /usr/bin/tini -w -e 143 -- java -Xms3g -Xmx3g -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom -XX:+ExitOnOutOfMemoryError -XX:MinRAMPercentage=15 -XX:MaxRAMPercentage=20 -classpath :lib/io.strimzi.cluster-operator-0.39.0.jar:lib/io.strimzi.api-0.39.0.jar:lib/io.fabric8.kubernetes-model-apiextensions-6.9.2.jar:lib/io.strimzi.config-model-0.39.0.jar:lib/io.strimzi.operator-common-0.39.0.jar:lib/io.strimzi.crd-annotations-0.39.0.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar:lib/org.eclipse.jetty.jetty-server-9.4.53.v20231009.jar:lib/org.eclipse.jetty.jetty-http-9.4.53.v20231009.jar:lib/org.eclipse.jetty.jetty-io-9.4.53.v20231009.jar:lib/javax.servlet.javax.servlet-api-3.1.0.jar:lib/io.fabric8.openshift-client-6.9.2.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.fabric8.openshift-client-api-6.9.2.jar:lib/io.fabric8.openshift-model-clusterautoscaling-6.9.2.jar:lib/io.fabric8.openshift-model-operator-6.9.2.jar:lib/io.fabric8.openshift-model-operatorhub-6.9.2.jar:lib/io.fabric8.openshift-model-machine-6.9.2.jar:lib/io.fabric8.openshift-model-whereabouts-6.9.2.jar:lib/io.fabric8.openshift-model-monitoring-6.9.2.jar:lib/io.fabric8.openshift-model-storageversionmigrator-6.9.2.jar:lib/io.fabric8.openshift-model-tuned-6.9.2.jar:lib/io.fabric8.openshift-model-console-6.9.2.jar:lib/io.fabric8.openshift-model-config-6.9.2.jar:lib/io.fabric8.openshift-model-machineconfig-6.9.2.jar:lib/io.fabric8.openshift-model-miscellaneous-6.9.2.jar:lib/io.fabric8.openshift-model-hive-6.9.2.jar:lib/io.fabric8.openshift-model-installer-6.9.2.jar:lib/io.fabric8.kubernetes-client-6.9.2.jar:lib/io.fabric8.kubernetes-httpclient-jdk-6.9.2.jar:lib/io.fabric8.kubernetes-client-api-6.9.2.jar:lib/io.fabric8.kubernetes-model-gatewayapi-6.9.2.jar:lib/io.fabric8.kubernetes-model-resource-6.9.2.jar:lib/io.fabric8.kubernetes-model-admissionregistration-6.9.2.jar:lib/io.fabric8.kubernetes-model-autoscaling-6.9.2.jar:lib/io.fabric8.kubernetes-model-batch-6.9.2.jar:lib/io.fabric8.kubernetes-model-certificates-6.9.2.jar:lib/io.fabric8.kubernetes-model-discovery-6.9.2.jar:lib/io.fabric8.kubernetes-model-extensions-6.9.2.jar:lib/io.fabric8.kubernetes-model-flowcontrol-6.9.2.jar:lib/io.fabric8.kubernetes-model-metrics-6.9.2.jar:lib/io.fabric8.kubernetes-model-scheduling-6.9.2.jar:lib/io.fabric8.kubernetes-model-node-6.9.2.jar:lib/org.slf4j.slf4j-api-1.7.36.jar:lib/org.snakeyaml.snakeyaml-engine-2.7.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.15.3.jar:lib/io.fabric8.kubernetes-model-core-6.9.2.jar:lib/io.fabric8.kubernetes-model-common-6.9.2.jar:lib/io.fabric8.kubernetes-model-events-6.9.2.jar:lib/io.fabric8.kubernetes-model-policy-6.9.2.jar:lib/io.fabric8.kubernetes-model-rbac-6.9.2.jar:lib/io.fabric8.kubernetes-model-apps-6.9.2.jar:lib/io.fabric8.kubernetes-model-storageclass-6.9.2.jar:lib/io.fabric8.kubernetes-model-networking-6.9.2.jar:lib/io.fabric8.kubernetes-model-coordination-6.9.2.jar:lib/io.fabric8.openshift-model-6.9.2.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.15.3.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.15.3.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.15.3.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.15.3.jar:lib/org.yaml.snakeyaml-2.1.jar:lib/io.vertx.vertx-core-4.5.0.jar:lib/io.netty.netty-common-4.1.100.Final.jar:lib/io.netty.netty-buffer-4.1.100.Final.jar:lib/io.netty.netty-handler-4.1.100.Final.jar:lib/io.netty.netty-handler-proxy-4.1.100.Final.jar:lib/io.netty.netty-codec-http-4.1.100.Final.jar:lib/io.netty.netty-codec-http2-4.1.100.Final.jar:lib/io.netty.netty-resolver-4.1.100.Final.jar:lib/io.netty.netty-resolver-dns-4.1.100.Final.jar:lib/io.netty.netty-codec-dns-4.1.100.Final.jar:lib/io.strimzi.certificate-manager-0.39.0.jar:lib/org.apache.logging.log4j.log4j-core-2.17.2.jar:lib/org.apache.logging.log4j.log4j-api-2.17.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar:lib/io.strimzi.kafka-oauth-server-0.14.0.jar:lib/io.strimzi.kafka-oauth-server-plain-0.14.0.jar:lib/io.strimzi.kafka-oauth-client-0.14.0.jar:lib/io.strimzi.kafka-oauth-common-0.14.0.jar:lib/com.nimbusds.nimbus-jose-jwt-9.31.jar:lib/com.github.stephenc.jcip.jcip-annotations-1.0-1.jar:lib/com.jayway.jsonpath.json-path-2.8.0.jar:lib/net.minidev.json-smart-2.4.10.jar:lib/net.minidev.accessors-smart-2.4.9.jar:lib/org.eclipse.jetty.jetty-util-9.4.53.v20231009.jar:lib/org.apache.kafka.kafka-clients-3.6.1.jar:lib/com.github.luben.zstd-jni-1.5.5-1.jar:lib/org.lz4.lz4-java-1.8.0.jar:lib/org.xerial.snappy.snappy-java-1.1.10.5.jar:lib/org.apache.kafka.kafka-server-common-3.6.1.jar:lib/com.yammer.metrics.metrics-core-2.2.0.jar:lib/net.sf.jopt-simple.jopt-simple-5.0.4.jar:lib/org.pcollections.pcollections-4.0.1.jar:lib/org.apache.zookeeper.zookeeper-3.8.3.jar:lib/org.apache.yetus.audience-annotations-0.12.0.jar:lib/commons-io.commons-io-2.11.0.jar:lib/org.apache.zookeeper.zookeeper-jute-3.8.3.jar:lib/io.netty.netty-transport-4.1.100.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.100.Final-linux-x86_64.jar:lib/io.netty.netty-transport-native-unix-common-4.1.100.Final.jar:lib/io.netty.netty-transport-classes-epoll-4.1.100.Final.jar:lib/io.micrometer.micrometer-core-1.9.5.jar:lib/org.hdrhistogram.HdrHistogram-2.1.12.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.micrometer.micrometer-registry-prometheus-1.9.5.jar:lib/io.prometheus.simpleclient_common-0.15.0.jar:lib/io.prometheus.simpleclient-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_common-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel_agent-0.15.0.jar:lib/io.vertx.vertx-micrometer-metrics-4.5.0.jar:lib/io.netty.netty-codec-4.1.100.Final.jar:lib/io.netty.netty-codec-socks-4.1.100.Final.jar io.strimzi.operator.cluster.Main
2024-02-07 10:31:02 INFO  Main:72 - ClusterOperator 0.39.0 is starting
2024-02-07 10:31:02 INFO  Util:239 - Using config:
	INGRESS_NGINX_CONTROLLER_PORT_80_TCP_PORT: 80
	PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
	container: oci
	CERT_MANAGER_WEBHOOK_PORT_443_TCP_PORT: 443
	STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE: quay.io/strimzi/operator:0.39.0
	INGRESS_NGINX_CONTROLLER_PORT: tcp://172.20.20.240:80
	INGRESS_NGINX_CONTROLLER_PORT_80_TCP_PROTO: tcp
	CERT_MANAGER_SERVICE_HOST: 172.20.8.129
	KUBE_DNS_PORT_53_UDP_PROTO: udp
	AWS_LOAD_BALANCER_WEBHOOK_SERVICE_SERVICE_PORT: 443
	KUBE_DNS_SERVICE_HOST: 172.20.0.10
	KUBERNETES_REQUEST_TIMEOUT: 90000
	INGRESS_NGINX_CONTROLLER_PORT_443_TCP_ADDR: 172.20.20.240
	INGRESS_NGINX_CONTROLLER_METRICS_PORT_10254_TCP_PORT: 10254
	KUBERNETES_CONNECTION_TIMEOUT: 90000
	INGRESS_NGINX_CONTROLLER_SERVICE_HOST: 172.20.20.240
	STRIMZI_HOME: /opt/strimzi
	PWD: /opt/strimzi
	JAVA_MAIN: io.strimzi.operator.cluster.Main
	CERT_MANAGER_WEBHOOK_SERVICE_PORT: 443
	STRIMZI_NAMESPACE: nexla,kube-system
	STRIMZI_DEFAULT_KANIKO_EXECUTOR_IMAGE: quay.io/strimzi/kaniko-executor:0.39.0
	KUBE_DNS_PORT_53_UDP: udp://172.20.0.10:53
	TINI_SHA256_S390X: 931b70a182af879ca249ae9de87ef68423121b38d235c78997fafc680ceab32d
	CERT_MANAGER_SERVICE_PORT: 9402
	STRIMZI_FEATURE_GATES: +UnidirectionalTopicOperator
	AWS_LOAD_BALANCER_WEBHOOK_SERVICE_SERVICE_HOST: 172.20.242.234
	KUBE_DNS_SERVICE_PORT_DNS: 53
	CERT_MANAGER_PORT_9402_TCP_PORT: 9402
	AWS_LOAD_BALANCER_WEBHOOK_SERVICE_PORT_443_TCP_PORT: 443
	INGRESS_NGINX_CONTROLLER_METRICS_SERVICE_HOST: 172.20.228.97
	CERT_MANAGER_WEBHOOK_SERVICE_HOST: 172.20.51.177
	KUBERNETES_WEBSOCKET_TIMEOUT: 50000
	CERT_MANAGER_WEBHOOK_PORT_443_TCP_PROTO: tcp
	AWS_LOAD_BALANCER_WEBHOOK_SERVICE_PORT_443_TCP_PROTO: tcp
	KUBERNETES_SERVICE_PORT_HTTPS: 443
	KUBE_DNS_PORT_53_UDP_ADDR: 172.20.0.10
	SHLVL: 0
	INGRESS_NGINX_CONTROLLER_SERVICE_PORT_HTTPS: 443
	KUBERNETES_PORT: tcp://172.20.0.1:443
	STRIMZI_KAFKA_MIRROR_MAKER_IMAGES: 3.5.0=quay.io/strimzi/kafka:0.39.0-kafka-3.5.0
3.5.1=quay.io/strimzi/kafka:0.39.0-kafka-3.5.1
3.5.2=quay.io/strimzi/kafka:0.39.0-kafka-3.5.2
3.6.0=quay.io/strimzi/kafka:0.39.0-kafka-3.6.0
3.6.1=quay.io/strimzi/kafka:0.39.0-kafka-3.6.1

	KUBE_DNS_PORT_53_TCP: tcp://172.20.0.10:53
	JAVA_HOME: /usr/lib/jvm/jre-17
	INGRESS_NGINX_CONTROLLER_METRICS_PORT: tcp://172.20.228.97:10254
	STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 3600000
	CERT_MANAGER_PORT_9402_TCP: tcp://172.20.8.129:9402
	JAVA_CLASSPATH: :lib/io.strimzi.cluster-operator-0.39.0.jar:lib/io.strimzi.api-0.39.0.jar:lib/io.fabric8.kubernetes-model-apiextensions-6.9.2.jar:lib/io.strimzi.config-model-0.39.0.jar:lib/io.strimzi.operator-common-0.39.0.jar:lib/io.strimzi.crd-annotations-0.39.0.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar:lib/org.eclipse.jetty.jetty-server-9.4.53.v20231009.jar:lib/org.eclipse.jetty.jetty-http-9.4.53.v20231009.jar:lib/org.eclipse.jetty.jetty-io-9.4.53.v20231009.jar:lib/javax.servlet.javax.servlet-api-3.1.0.jar:lib/io.fabric8.openshift-client-6.9.2.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.fabric8.openshift-client-api-6.9.2.jar:lib/io.fabric8.openshift-model-clusterautoscaling-6.9.2.jar:lib/io.fabric8.openshift-model-operator-6.9.2.jar:lib/io.fabric8.openshift-model-operatorhub-6.9.2.jar:lib/io.fabric8.openshift-model-machine-6.9.2.jar:lib/io.fabric8.openshift-model-whereabouts-6.9.2.jar:lib/io.fabric8.openshift-model-monitoring-6.9.2.jar:lib/io.fabric8.openshift-model-storageversionmigrator-6.9.2.jar:lib/io.fabric8.openshift-model-tuned-6.9.2.jar:lib/io.fabric8.openshift-model-console-6.9.2.jar:lib/io.fabric8.openshift-model-config-6.9.2.jar:lib/io.fabric8.openshift-model-machineconfig-6.9.2.jar:lib/io.fabric8.openshift-model-miscellaneous-6.9.2.jar:lib/io.fabric8.openshift-model-hive-6.9.2.jar:lib/io.fabric8.openshift-model-installer-6.9.2.jar:lib/io.fabric8.kubernetes-client-6.9.2.jar:lib/io.fabric8.kubernetes-httpclient-jdk-6.9.2.jar:lib/io.fabric8.kubernetes-client-api-6.9.2.jar:lib/io.fabric8.kubernetes-model-gatewayapi-6.9.2.jar:lib/io.fabric8.kubernetes-model-resource-6.9.2.jar:lib/io.fabric8.kubernetes-model-admissionregistration-6.9.2.jar:lib/io.fabric8.kubernetes-model-autoscaling-6.9.2.jar:lib/io.fabric8.kubernetes-model-batch-6.9.2.jar:lib/io.fabric8.kubernetes-model-certificates-6.9.2.jar:lib/io.fabric8.kubernetes-model-discovery-6.9.2.jar:lib/io.fabric8.kubernetes-model-extensions-6.9.2.jar:lib/io.fabric8.kubernetes-model-flowcontrol-6.9.2.jar:lib/io.fabric8.kubernetes-model-metrics-6.9.2.jar:lib/io.fabric8.kubernetes-model-scheduling-6.9.2.jar:lib/io.fabric8.kubernetes-model-node-6.9.2.jar:lib/org.slf4j.slf4j-api-1.7.36.jar:lib/org.snakeyaml.snakeyaml-engine-2.7.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.15.3.jar:lib/io.fabric8.kubernetes-model-core-6.9.2.jar:lib/io.fabric8.kubernetes-model-common-6.9.2.jar:lib/io.fabric8.kubernetes-model-events-6.9.2.jar:lib/io.fabric8.kubernetes-model-policy-6.9.2.jar:lib/io.fabric8.kubernetes-model-rbac-6.9.2.jar:lib/io.fabric8.kubernetes-model-apps-6.9.2.jar:lib/io.fabric8.kubernetes-model-storageclass-6.9.2.jar:lib/io.fabric8.kubernetes-model-networking-6.9.2.jar:lib/io.fabric8.kubernetes-model-coordination-6.9.2.jar:lib/io.fabric8.openshift-model-6.9.2.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.15.3.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.15.3.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.15.3.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.15.3.jar:lib/org.yaml.snakeyaml-2.1.jar:lib/io.vertx.vertx-core-4.5.0.jar:lib/io.netty.netty-common-4.1.100.Final.jar:lib/io.netty.netty-buffer-4.1.100.Final.jar:lib/io.netty.netty-handler-4.1.100.Final.jar:lib/io.netty.netty-handler-proxy-4.1.100.Final.jar:lib/io.netty.netty-codec-http-4.1.100.Final.jar:lib/io.netty.netty-codec-http2-4.1.100.Final.jar:lib/io.netty.netty-resolver-4.1.100.Final.jar:lib/io.netty.netty-resolver-dns-4.1.100.Final.jar:lib/io.netty.netty-codec-dns-4.1.100.Final.jar:lib/io.strimzi.certificate-manager-0.39.0.jar:lib/org.apache.logging.log4j.log4j-core-2.17.2.jar:lib/org.apache.logging.log4j.log4j-api-2.17.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar:lib/io.strimzi.kafka-oauth-server-0.14.0.jar:lib/io.strimzi.kafka-oauth-server-plain-0.14.0.jar:lib/io.strimzi.kafka-oauth-client-0.14.0.jar:lib/io.strimzi.kafka-oauth-common-0.14.0.jar:lib/com.nimbusds.nimbus-jose-jwt-9.31.jar:lib/com.github.stephenc.jcip.jcip-annotations-1.0-1.jar:lib/com.jayway.jsonpath.json-path-2.8.0.jar:lib/net.minidev.json-smart-2.4.10.jar:lib/net.minidev.accessors-smart-2.4.9.jar:lib/org.eclipse.jetty.jetty-util-9.4.53.v20231009.jar:lib/org.apache.kafka.kafka-clients-3.6.1.jar:lib/com.github.luben.zstd-jni-1.5.5-1.jar:lib/org.lz4.lz4-java-1.8.0.jar:lib/org.xerial.snappy.snappy-java-1.1.10.5.jar:lib/org.apache.kafka.kafka-server-common-3.6.1.jar:lib/com.yammer.metrics.metrics-core-2.2.0.jar:lib/net.sf.jopt-simple.jopt-simple-5.0.4.jar:lib/org.pcollections.pcollections-4.0.1.jar:lib/org.apache.zookeeper.zookeeper-3.8.3.jar:lib/org.apache.yetus.audience-annotations-0.12.0.jar:lib/commons-io.commons-io-2.11.0.jar:lib/org.apache.zookeeper.zookeeper-jute-3.8.3.jar:lib/io.netty.netty-transport-4.1.100.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.100.Final-linux-x86_64.jar:lib/io.netty.netty-transport-native-unix-common-4.1.100.Final.jar:lib/io.netty.netty-transport-classes-epoll-4.1.100.Final.jar:lib/io.micrometer.micrometer-core-1.9.5.jar:lib/org.hdrhistogram.HdrHistogram-2.1.12.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.micrometer.micrometer-registry-prometheus-1.9.5.jar:lib/io.prometheus.simpleclient_common-0.15.0.jar:lib/io.prometheus.simpleclient-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_common-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel_agent-0.15.0.jar:lib/io.vertx.vertx-micrometer-metrics-4.5.0.jar:lib/io.netty.netty-codec-4.1.100.Final.jar:lib/io.netty.netty-codec-socks-4.1.100.Final.jar
	INGRESS_NGINX_CONTROLLER_METRICS_PORT_10254_TCP: tcp://172.20.228.97:10254
	INGRESS_NGINX_CONTROLLER_SERVICE_PORT_HTTP: 80
	STRIMZI_DEFAULT_CRUISE_CONTROL_IMAGE: quay.io/strimzi/kafka:0.39.0-kafka-3.6.1
	CERT_MANAGER_WEBHOOK_PORT: tcp://172.20.51.177:443
	KUBERNETES_PORT_443_TCP_ADDR: 172.20.0.1
	CERT_MANAGER_WEBHOOK_SERVICE_PORT_HTTPS: 443
	CERT_MANAGER_WEBHOOK_PORT_443_TCP: tcp://172.20.51.177:443
	STRIMZI_LEADER_ELECTION_LEASE_NAMESPACE: kube-system
	STRIMZI_ZOOKEEPER_SESSION_TIMEOUT_MS: 900000
	KUBE_DNS_SERVICE_PORT_DNS_TCP: 53
	STRIMZI_KAFKA_MIRROR_MAKER_2_IMAGES: 3.5.0=quay.io/strimzi/kafka:0.39.0-kafka-3.5.0
3.5.1=quay.io/strimzi/kafka:0.39.0-kafka-3.5.1
3.5.2=quay.io/strimzi/kafka:0.39.0-kafka-3.5.2
3.6.0=quay.io/strimzi/kafka:0.39.0-kafka-3.6.0
3.6.1=quay.io/strimzi/kafka:0.39.0-kafka-3.6.1

	KUBERNETES_PORT_443_TCP_PROTO: tcp
	STRIMZI_DEFAULT_USER_OPERATOR_IMAGE: quay.io/strimzi/operator:0.39.0
	KUBE_DNS_PORT_53_TCP_PORT: 53
	AWS_LOAD_BALANCER_WEBHOOK_SERVICE_PORT: tcp://172.20.242.234:443
	STRIMZI_DEFAULT_KAFKA_INIT_IMAGE: quay.io/strimzi/operator:0.39.0
	INGRESS_NGINX_CONTROLLER_PORT_80_TCP: tcp://172.20.20.240:80
	STRIMZI_KAFKA_IMAGES: 3.5.0=quay.io/strimzi/kafka:0.39.0-kafka-3.5.0
3.5.1=quay.io/strimzi/kafka:0.39.0-kafka-3.5.1
3.5.2=quay.io/strimzi/kafka:0.39.0-kafka-3.5.2
3.6.0=quay.io/strimzi/kafka:0.39.0-kafka-3.6.0
3.6.1=quay.io/strimzi/kafka:0.39.0-kafka-3.6.1

	STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE: quay.io/strimzi/kafka:0.39.0-kafka-3.6.1
	KUBE_DNS_SERVICE_PORT: 53
	KUBE_DNS_PORT_53_UDP_PORT: 53
	INGRESS_NGINX_CONTROLLER_PORT_443_TCP_PORT: 443
	JAVA_OPTS: -Xms3g -Xmx3g -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom  -XX:+ExitOnOutOfMemoryError -XX:MinRAMPercentage=15 -XX:MaxRAMPercentage=20
	CERT_MANAGER_WEBHOOK_PORT_443_TCP_ADDR: 172.20.51.177
	INGRESS_NGINX_CONTROLLER_PORT_443_TCP: tcp://172.20.20.240:443
	INGRESS_NGINX_CONTROLLER_METRICS_PORT_10254_TCP_ADDR: 172.20.228.97
	INGRESS_NGINX_CONTROLLER_PORT_80_TCP_ADDR: 172.20.20.240
	KUBE_DNS_PORT_53_TCP_ADDR: 172.20.0.10
	AWS_LOAD_BALANCER_WEBHOOK_SERVICE_PORT_443_TCP_ADDR: 172.20.242.234
	STRIMZI_LEADER_ELECTION_IDENTITY: strimzi-cluster-operator-854c84cd85-l2n5d
	KUBERNETES_PORT_443_TCP: tcp://172.20.0.1:443
	STRIMZI_LEADER_ELECTION_ENABLED: true
	STRIMZI_VERSION: 0.39.0
	INGRESS_NGINX_CONTROLLER_METRICS_SERVICE_PORT_METRICS: 10254
	STRIMZI_DEFAULT_MAVEN_BUILDER: quay.io/strimzi/maven-builder:0.39.0
	CERT_MANAGER_PORT: tcp://172.20.8.129:9402
	AWS_LOAD_BALANCER_WEBHOOK_SERVICE_SERVICE_PORT_WEBHOOK_SERVER: 443
	TINI_SHA256_AMD64: 93dcc18adc78c65a028a84799ecf8ad40c936fdfc5f2a57b1acda5a8117fa82c
	KUBERNETES_SERVICE_DNS_DOMAIN: cluster.local
	CERT_MANAGER_PORT_9402_TCP_PROTO: tcp
	KUBE_DNS_PORT: udp://172.20.0.10:53
	CERT_MANAGER_PORT_9402_TCP_ADDR: 172.20.8.129
	STRIMZI_LEADER_ELECTION_LEASE_NAME: strimzi-cluster-operator
	CERT_MANAGER_SERVICE_PORT_TCP_PROMETHEUS_SERVICEMONITOR: 9402
	INGRESS_NGINX_CONTROLLER_SERVICE_PORT: 80
	STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE: quay.io/strimzi/kafka-bridge:0.27.0
	KUBERNETES_SERVICE_HOST: 172.20.0.1
	INGRESS_NGINX_CONTROLLER_METRICS_PORT_10254_TCP_PROTO: tcp
	KUBE_DNS_PORT_53_TCP_PROTO: tcp
	TINI_VERSION: v0.19.0
	STRIMZI_OPERATION_TIMEOUT_MS: 1800000
	KUBERNETES_SERVICE_PORT: 443
	INGRESS_NGINX_CONTROLLER_PORT_443_TCP_PROTO: tcp
	INGRESS_NGINX_CONTROLLER_METRICS_SERVICE_PORT: 10254
	STRIMZI_DEFAULT_KAFKA_EXPORTER_IMAGE: quay.io/strimzi/kafka:0.39.0-kafka-3.6.1
	TINI_SHA256_ARM64: 07952557df20bfd2a95f9bef198b445e006171969499a1d361bd9e6f8e5e0e81
	TINI_SHA256_PPC64LE: 3f658420974768e40810001a038c29d003728c5fe86da211cff5059e48cfdfde
	HOSTNAME: strimzi-cluster-operator-854c84cd85-l2n5d
	STRIMZI_KAFKA_CONNECT_IMAGES: 3.5.0=quay.io/strimzi/kafka:0.39.0-kafka-3.5.0
3.5.1=quay.io/strimzi/kafka:0.39.0-kafka-3.5.1
3.5.2=quay.io/strimzi/kafka:0.39.0-kafka-3.5.2
3.6.0=quay.io/strimzi/kafka:0.39.0-kafka-3.6.0
3.6.1=quay.io/strimzi/kafka:0.39.0-kafka-3.6.1

	KUBERNETES_PORT_443_TCP_PORT: 443
	STRIMZI_OPERATOR_NAME: strimzi-cluster-operator-854c84cd85-l2n5d
	HOME: /home/strimzi
	MALLOC_ARENA_MAX: 2
	AWS_LOAD_BALANCER_WEBHOOK_SERVICE_PORT_443_TCP: tcp://172.20.242.234:443
	STRIMZI_OPERATOR_NAMESPACE: kube-system

2024-02-07 10:31:02 INFO  Main:75 - Cluster Operator configuration is ClusterOperatorConfig{
	namespaces='[kube-system, nexla]'
	reconciliationIntervalMs=3600000
	operationTimeoutMs=1800000
	connectBuildTimeoutMs=300000
	createClusterRoles=false
	networkPolicyGeneration=true
	versions='versions{3.5.0={proto: 3.5 msg: 3.5 kafka-image: quay.io/strimzi/kafka:0.39.0-kafka-3.5.0 connect-image: quay.io/strimzi/kafka:0.39.0-kafka-3.5.0 mirrormaker-image: quay.io/strimzi/kafka:0.39.0-kafka-3.5.0 mirrormaker2-image: quay.io/strimzi/kafka:0.39.0-kafka-3.5.0}, 3.5.1={proto: 3.5 msg: 3.5 kafka-image: quay.io/strimzi/kafka:0.39.0-kafka-3.5.1 connect-image: quay.io/strimzi/kafka:0.39.0-kafka-3.5.1 mirrormaker-image: quay.io/strimzi/kafka:0.39.0-kafka-3.5.1 mirrormaker2-image: quay.io/strimzi/kafka:0.39.0-kafka-3.5.1}, 3.5.2={proto: 3.5 msg: 3.5 kafka-image: quay.io/strimzi/kafka:0.39.0-kafka-3.5.2 connect-image: quay.io/strimzi/kafka:0.39.0-kafka-3.5.2 mirrormaker-image: quay.io/strimzi/kafka:0.39.0-kafka-3.5.2 mirrormaker2-image: quay.io/strimzi/kafka:0.39.0-kafka-3.5.2}, 3.6.0={proto: 3.6 msg: 3.6 kafka-image: quay.io/strimzi/kafka:0.39.0-kafka-3.6.0 connect-image: quay.io/strimzi/kafka:0.39.0-kafka-3.6.0 mirrormaker-image: quay.io/strimzi/kafka:0.39.0-kafka-3.6.0 mirrormaker2-image: quay.io/strimzi/kafka:0.39.0-kafka-3.6.0}, 3.6.1={proto: 3.6 msg: 3.6 kafka-image: quay.io/strimzi/kafka:0.39.0-kafka-3.6.1 connect-image: quay.io/strimzi/kafka:0.39.0-kafka-3.6.1 mirrormaker-image: quay.io/strimzi/kafka:0.39.0-kafka-3.6.1 mirrormaker2-image: quay.io/strimzi/kafka:0.39.0-kafka-3.6.1}}'
	imagePullPolicy='null'
	imagePullSecrets='null'
	operatorNamespace='kube-system'
	operatorNamespaceLabels='null'
	customResourceSelector='null'
	featureGates='FeatureGates(UseKRaft=false,KafkaNodePools=true,UnidirectionalTopicOperator=true)'
	zkAdminSessionTimeoutMs=10000
	dnsCacheTtlSec=30
	podSetReconciliationOnly=false
	podSetControllerWorkQueueSize=1024
	operatorName='strimzi-cluster-operator-854c84cd85-l2n5d'
	podSecurityProviderClass='io.strimzi.plugin.security.profiles.impl.BaselinePodSecurityProvider'
	leaderElectionConfig='LeaderElectionConfig{leaseName='strimzi-cluster-operator', namespace='kube-system', identity='strimzi-cluster-operator-854c84cd85-l2n5d', leaseDuration=PT15S, renewDeadline=PT10S, retryPeriod=PT2S}'}
2024-02-07 10:31:03 INFO  Main:345 - Health and metrics server is ready on port 8080)
2024-02-07 10:31:03 INFO  Main:255 - Waiting to become a leader
2024-02-07 10:31:03 INFO  LeaderElectionManager:69 - Starting the Leader Elector
2024-02-07 10:31:03 INFO  LeaderElectionManager:121 - The new leader is
2024-02-07 10:31:03 INFO  LeaderElectionManager:121 - The new leader is strimzi-cluster-operator-854c84cd85-l2n5d
2024-02-07 10:31:03 INFO  LeaderElectionManager:103 - Started being a leader
2024-02-07 10:31:03 INFO  Main:237 - I'm the new leader
2024-02-07 10:31:03 WARN  PlatformFeaturesAvailability:156 - API Group route.openshift.io is not supported
2024-02-07 10:31:03 WARN  PlatformFeaturesAvailability:156 - API Group build.openshift.io is not supported
2024-02-07 10:31:03 WARN  PlatformFeaturesAvailability:156 - API Group image.openshift.io is not supported
2024-02-07 10:31:03 INFO  Main:124 - Environment facts gathered: PlatformFeaturesAvailability(KubernetesVersion=1.28,OpenShiftRoutes=false,OpenShiftBuilds=false,OpenShiftImageStreams=false)
2024-02-07 10:31:03 INFO  PodSecurityProviderFactory:43 - Found PodSecurityProvider io.strimzi.plugin.security.profiles.impl.BaselinePodSecurityProvider
2024-02-07 10:31:03 INFO  PodSecurityProviderFactory:62 - Initializing PodSecurityProvider io.strimzi.plugin.security.profiles.impl.BaselinePodSecurityProvider
2024-02-07 10:31:03 INFO  ClusterOperator:86 - Creating ClusterOperator for namespace kube-system
2024-02-07 10:31:03 INFO  ClusterOperator:100 - Starting ClusterOperator for namespace kube-system
2024-02-07 10:31:03 INFO  ClusterOperator:86 - Creating ClusterOperator for namespace nexla
2024-02-07 10:31:03 INFO  ClusterOperator:100 - Starting ClusterOperator for namespace nexla
2024-02-07 10:31:03 INFO  StrimziPodSetController:583 - Starting the StrimziPodSet controller
2024-02-07 10:31:03 INFO  StrimziPodSetController:583 - Starting the StrimziPodSet controller
2024-02-07 10:31:03 INFO  StrimziPodSetController:552 - Starting StrimziPodSet controller for namespace nexla
2024-02-07 10:31:03 INFO  StrimziPodSetController:552 - Starting StrimziPodSet controller for namespace kube-system
2024-02-07 10:31:03 WARN  VersionUsageUtils:60 - The client is using resource type 'strimzipodsets' with unstable version 'v1beta2'
2024-02-07 10:31:03 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkas' with unstable version 'v1beta2'
2024-02-07 10:31:04 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkaconnects' with unstable version 'v1beta2'
2024-02-07 10:31:04 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkamirrormaker2s' with unstable version 'v1beta2'
2024-02-07 10:31:04 INFO  StrimziPodSetController:555 - Waiting for informers to sync
2024-02-07 10:31:04 INFO  StrimziPodSetController:555 - Waiting for informers to sync
2024-02-07 10:31:04 INFO  ClusterOperator:114 - Opened watch for Kafka operator
2024-02-07 10:31:04 INFO  ClusterOperator:114 - Opened watch for KafkaMirrorMaker operator
2024-02-07 10:31:04 INFO  ClusterOperator:114 - Opened watch for KafkaConnect operator
2024-02-07 10:31:04 INFO  ClusterOperator:114 - Opened watch for KafkaBridge operator
2024-02-07 10:31:04 INFO  ClusterOperator:114 - Opened watch for KafkaMirrorMaker2 operator
2024-02-07 10:31:04 INFO  ClusterOperator:114 - Opened watch for KafkaRebalance operator
2024-02-07 10:31:04 INFO  ClusterOperator:123 - Opened watch for KafkaNodePool operator
2024-02-07 10:31:04 INFO  ClusterOperator:131 - Opened watch for KafkaConnector operator
2024-02-07 10:31:04 INFO  ClusterOperator:139 - Setting up periodic reconciliation for namespace kube-system
2024-02-07 10:31:04 INFO  Main:206 - Cluster Operator verticle started in namespace kube-system without label selector
2024-02-07 10:31:05 INFO  ClusterOperator:114 - Opened watch for Kafka operator
2024-02-07 10:31:05 INFO  AbstractOperator:511 - Reconciliation #1(watch) Kafka(nexla/kafka-cluster): Kafka kafka-cluster in namespace nexla was ADDED
2024-02-07 10:31:05 INFO  ClusterOperator:114 - Opened watch for KafkaMirrorMaker operator
2024-02-07 10:31:05 INFO  StrimziPodSetController:560 - Informers are in-sync
2024-02-07 10:31:05 INFO  StrimziPodSetController:560 - Informers are in-sync
2024-02-07 10:31:05 INFO  ClusterOperator:114 - Opened watch for KafkaConnect operator
2024-02-07 10:31:05 INFO  ClusterOperator:114 - Opened watch for KafkaBridge operator
2024-02-07 10:31:05 INFO  ClusterOperator:114 - Opened watch for KafkaMirrorMaker2 operator
2024-02-07 10:31:05 INFO  AbstractOperator:265 - Reconciliation #1(watch) Kafka(nexla/kafka-cluster): Kafka kafka-cluster will be checked for creation or modification
2024-02-07 10:31:05 INFO  ClusterOperator:114 - Opened watch for KafkaRebalance operator
2024-02-07 10:31:05 INFO  ClusterOperator:123 - Opened watch for KafkaNodePool operator
2024-02-07 10:31:05 INFO  ClusterOperator:131 - Opened watch for KafkaConnector operator
2024-02-07 10:31:05 INFO  ClusterOperator:139 - Setting up periodic reconciliation for namespace nexla
2024-02-07 10:31:05 INFO  Main:206 - Cluster Operator verticle started in namespace nexla without label selector
2024-02-07 10:31:56 WARN  KafkaAvailability:90 - Reconciliation #1(watch) Kafka(nexla/kafka-cluster): Error determining whether it is safe to restart pod 0
java.util.concurrent.CompletionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=describeConfigs, deadlineMs=1707301916589, tries=2, nextAllowedTryMs=1707301916707) timed out at 1707301916607 after 2 attempt(s)
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
	at org.apache.kafka.common.internals.KafkaCompletableFuture.kafkaCompleteExceptionally(KafkaCompletableFuture.java:49) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl.completeExceptionally(KafkaFutureImpl.java:130) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.common.KafkaFuture.lambda$allOf$2(KafkaFuture.java:93) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
	at org.apache.kafka.common.internals.KafkaCompletableFuture.kafkaCompleteExceptionally(KafkaCompletableFuture.java:49) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl.completeExceptionally(KafkaFutureImpl.java:130) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient.lambda$completeAllExceptionally$1(KafkaAdminClient.java:406) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1779) ~[?:?]
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762) ~[?:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient.completeAllExceptionally(KafkaAdminClient.java:406) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient.completeAllExceptionally(KafkaAdminClient.java:395) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient.access$3000(KafkaAdminClient.java:284) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$11.handleFailure(KafkaAdminClient.java:2380) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$Call.handleTimeoutFailure(KafkaAdminClient.java:821) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$Call.fail(KafkaAdminClient.java:785) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.handleResponses(KafkaAdminClient.java:1230) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1389) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1312) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=describeConfigs, deadlineMs=1707301916589, tries=2, nextAllowedTryMs=1707301916707) timed out at 1707301916607 after 2 attempt(s)
Caused by: org.apache.kafka.common.errors.DisconnectException: Cancelled describeConfigs request with correlation id 10 due to node 3 being disconnected
2024-02-07 10:31:56 INFO  KafkaRoller:396 - Reconciliation #1(watch) Kafka(nexla/kafka-cluster): Will temporarily skip verifying pod kafka-cluster-kafka-0/0 is up-to-date due to ForceableProblem: An error while trying to determine the possibility of updating Kafka broker pods, retrying after at least 250ms
2024-02-07 10:32:05 INFO  AbstractOperator:400 - Reconciliation #1(watch) Kafka(nexla/kafka-cluster): Reconciliation is in progress
2024-02-07 10:32:40 WARN  KafkaAvailability:90 - Reconciliation #1(watch) Kafka(nexla/kafka-cluster): Error determining whether it is safe to restart pod 2
java.util.concurrent.CompletionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=describeConfigs, deadlineMs=1707301960911, tries=2, nextAllowedTryMs=1707301961025) timed out at 1707301960925 after 2 attempt(s)
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
	at org.apache.kafka.common.internals.KafkaCompletableFuture.kafkaCompleteExceptionally(KafkaCompletableFuture.java:49) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl.completeExceptionally(KafkaFutureImpl.java:130) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.common.KafkaFuture.lambda$allOf$2(KafkaFuture.java:93) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
	at org.apache.kafka.common.internals.KafkaCompletableFuture.kafkaCompleteExceptionally(KafkaCompletableFuture.java:49) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl.completeExceptionally(KafkaFutureImpl.java:130) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient.lambda$completeAllExceptionally$1(KafkaAdminClient.java:406) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1779) ~[?:?]
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762) ~[?:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient.completeAllExceptionally(KafkaAdminClient.java:406) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient.completeAllExceptionally(KafkaAdminClient.java:395) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient.access$3000(KafkaAdminClient.java:284) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$11.handleFailure(KafkaAdminClient.java:2380) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$Call.handleTimeoutFailure(KafkaAdminClient.java:821) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$Call.fail(KafkaAdminClient.java:785) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.handleResponses(KafkaAdminClient.java:1230) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1389) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1312) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=describeConfigs, deadlineMs=1707301960911, tries=2, nextAllowedTryMs=1707301961025) timed out at 1707301960925 after 2 attempt(s)
Caused by: org.apache.kafka.common.errors.DisconnectException: Cancelled describeConfigs request with correlation id 23 due to node 1 being disconnected
2024-02-07 10:32:40 INFO  KafkaRoller:396 - Reconciliation #1(watch) Kafka(nexla/kafka-cluster): Will temporarily skip verifying pod kafka-cluster-kafka-2/2 is up-to-date due to ForceableProblem: An error while trying to determine the possibility of updating Kafka broker pods, retrying after at least 250ms
2024-02-07 10:32:41 INFO  KafkaRoller:396 - Reconciliation #1(watch) Kafka(nexla/kafka-cluster): Will temporarily skip verifying pod kafka-cluster-kafka-3/3 is up-to-date due to ForceableProblem: Pod kafka-cluster-kafka-3 is the active controller and there are other pods to verify first, retrying after at least 250ms
2024-02-07 10:33:05 INFO  AbstractOperator:400 - Reconciliation #1(watch) Kafka(nexla/kafka-cluster): Reconciliation is in progress
2024-02-07 10:33:27 WARN  KafkaAvailability:90 - Reconciliation #1(watch) Kafka(nexla/kafka-cluster): Error determining whether it is safe to restart pod 0
java.util.concurrent.CompletionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=describeConfigs, deadlineMs=1707302007237, tries=2, nextAllowedTryMs=1707302007353) timed out at 1707302007253 after 2 attempt(s)
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
	at org.apache.kafka.common.internals.KafkaCompletableFuture.kafkaCompleteExceptionally(KafkaCompletableFuture.java:49) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl.completeExceptionally(KafkaFutureImpl.java:130) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.common.KafkaFuture.lambda$allOf$2(KafkaFuture.java:93) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
	at org.apache.kafka.common.internals.KafkaCompletableFuture.kafkaCompleteExceptionally(KafkaCompletableFuture.java:49) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl.completeExceptionally(KafkaFutureImpl.java:130) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient.lambda$completeAllExceptionally$1(KafkaAdminClient.java:406) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1779) ~[?:?]
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762) ~[?:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient.completeAllExceptionally(KafkaAdminClient.java:406) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient.completeAllExceptionally(KafkaAdminClient.java:395) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient.access$3000(KafkaAdminClient.java:284) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$11.handleFailure(KafkaAdminClient.java:2380) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$Call.handleTimeoutFailure(KafkaAdminClient.java:821) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$Call.fail(KafkaAdminClient.java:785) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.handleResponses(KafkaAdminClient.java:1230) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1389) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1312) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=describeConfigs, deadlineMs=1707302007237, tries=2, nextAllowedTryMs=1707302007353) timed out at 1707302007253 after 2 attempt(s)
Caused by: org.apache.kafka.common.errors.DisconnectException: Cancelled describeConfigs request with correlation id 38 due to node 3 being disconnected
2024-02-07 10:33:27 INFO  KafkaRoller:396 - Reconciliation #1(watch) Kafka(nexla/kafka-cluster): Will temporarily skip verifying pod kafka-cluster-kafka-0/0 is up-to-date due to ForceableProblem: An error while trying to determine the possibility of updating Kafka broker pods, retrying after at least 500ms
2024-02-07 10:34:05 INFO  AbstractOperator:400 - Reconciliation #1(watch) Kafka(nexla/kafka-cluster): Reconciliation is in progress
2024-02-07 10:34:14 WARN  KafkaAvailability:90 - Reconciliation #1(watch) Kafka(nexla/kafka-cluster): Error determining whether it is safe to restart pod 2
java.util.concurrent.CompletionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=describeConfigs, deadlineMs=1707302054164, tries=2, nextAllowedTryMs=1707302054281) timed out at 1707302054181 after 2 attempt(s)
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
	at org.apache.kafka.common.internals.KafkaCompletableFuture.kafkaCompleteExceptionally(KafkaCompletableFuture.java:49) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl.completeExceptionally(KafkaFutureImpl.java:130) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.common.KafkaFuture.lambda$allOf$2(KafkaFuture.java:93) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
	at org.apache.kafka.common.internals.KafkaCompletableFuture.kafkaCompleteExceptionally(KafkaCompletableFuture.java:49) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.common.internals.KafkaFutureImpl.completeExceptionally(KafkaFutureImpl.java:130) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient.lambda$completeAllExceptionally$1(KafkaAdminClient.java:406) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1779) ~[?:?]
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762) ~[?:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient.completeAllExceptionally(KafkaAdminClient.java:406) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient.completeAllExceptionally(KafkaAdminClient.java:395) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient.access$3000(KafkaAdminClient.java:284) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$11.handleFailure(KafkaAdminClient.java:2380) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$Call.handleTimeoutFailure(KafkaAdminClient.java:821) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$Call.fail(KafkaAdminClient.java:785) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.handleResponses(KafkaAdminClient.java:1230) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1389) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1312) ~[org.apache.kafka.kafka-clients-3.6.1.jar:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=describeConfigs, deadlineMs=1707302054164, tries=2, nextAllowedTryMs=1707302054281) timed out at 1707302054181 after 2 attempt(s)
Caused by: org.apache.kafka.common.errors.DisconnectException: Cancelled describeConfigs request with correlation id 46 due to node 4 being disconnected
2024-02-07 10:34:14 INFO  KafkaRoller:396 - Reconciliation #1(watch) Kafka(nexla/kafka-cluster): Will temporarily skip verifying pod kafka-cluster-kafka-2/2 is up-to-date due to ForceableProblem: An error while trying to determine the possibility of updating Kafka broker pods, retrying after at least 500ms
2024-02-07 10:34:14 INFO  KafkaRoller:396 - Reconciliation #1(watch) Kafka(nexla/kafka-cluster): Will temporarily skip verifying pod kafka-cluster-kafka-3/3 is up-to-date due to ForceableProblem: Pod kafka-cluster-kafka-3 is the active controller and there are other pods to verify first, retrying after at least 500ms

[sachintandon-nexla]

Strimzi operator Deployment definition

---
# Source: kafka-clusters/charts/strimzi-kafka-operator/templates/060-Deployment-strimzi-cluster-operator.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: strimzi-cluster-operator
  namespace: kube-system
  labels:
    app: strimzi
    chart: strimzi-kafka-operator-0.39.0
    component: deployment
    release: release-name
    heritage: Helm
spec:
  replicas: 1
  selector:
    matchLabels:
      name: strimzi-cluster-operator
      strimzi.io/kind: cluster-operator
  template:
    metadata:
      labels:
        name: strimzi-cluster-operator
        strimzi.io/kind: cluster-operator
    spec:
      serviceAccountName: strimzi-cluster-operator
      volumes:
        - name: strimzi-tmp
          emptyDir:
            medium: Memory
            sizeLimit: 1Mi
        - name: co-config-volume
          configMap:
            name: strimzi-cluster-operator
      containers:
        - name: strimzi-cluster-operator
          image: quay.io/strimzi/operator:0.39.0
          ports:
            - containerPort: 8080
              name: http
          args:
            - /opt/strimzi/bin/cluster_operator_run.sh
          volumeMounts:
            - name: strimzi-tmp
              mountPath: /tmp
            - name: co-config-volume
              mountPath: /opt/strimzi/custom-config/
          env:
            - name: STRIMZI_NAMESPACE
              value: "nexla,nexla"
            - name: STRIMZI_FULL_RECONCILIATION_INTERVAL_MS
              value: "3600000"
            - name: STRIMZI_OPERATION_TIMEOUT_MS
              value: "1800000"
            - name: STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE
              value: quay.io/strimzi/kafka:0.39.0-kafka-3.6.1
            - name: STRIMZI_DEFAULT_KAFKA_EXPORTER_IMAGE
              value: quay.io/strimzi/kafka:0.39.0-kafka-3.6.1
            - name: STRIMZI_DEFAULT_CRUISE_CONTROL_IMAGE
              value: quay.io/strimzi/kafka:0.39.0-kafka-3.6.1
            - name: STRIMZI_KAFKA_IMAGES
              value: |
                3.5.0=quay.io/strimzi/kafka:0.39.0-kafka-3.5.0
                3.5.1=quay.io/strimzi/kafka:0.39.0-kafka-3.5.1
                3.5.2=quay.io/strimzi/kafka:0.39.0-kafka-3.5.2
                3.6.0=quay.io/strimzi/kafka:0.39.0-kafka-3.6.0
                3.6.1=quay.io/strimzi/kafka:0.39.0-kafka-3.6.1
            - name: STRIMZI_KAFKA_CONNECT_IMAGES
              value: |
                3.5.0=quay.io/strimzi/kafka:0.39.0-kafka-3.5.0
                3.5.1=quay.io/strimzi/kafka:0.39.0-kafka-3.5.1
                3.5.2=quay.io/strimzi/kafka:0.39.0-kafka-3.5.2
                3.6.0=quay.io/strimzi/kafka:0.39.0-kafka-3.6.0
                3.6.1=quay.io/strimzi/kafka:0.39.0-kafka-3.6.1
            - name: STRIMZI_KAFKA_MIRROR_MAKER_IMAGES
              value: |
                3.5.0=quay.io/strimzi/kafka:0.39.0-kafka-3.5.0
                3.5.1=quay.io/strimzi/kafka:0.39.0-kafka-3.5.1
                3.5.2=quay.io/strimzi/kafka:0.39.0-kafka-3.5.2
                3.6.0=quay.io/strimzi/kafka:0.39.0-kafka-3.6.0
                3.6.1=quay.io/strimzi/kafka:0.39.0-kafka-3.6.1
            - name: STRIMZI_KAFKA_MIRROR_MAKER_2_IMAGES
              value: |
                3.5.0=quay.io/strimzi/kafka:0.39.0-kafka-3.5.0
                3.5.1=quay.io/strimzi/kafka:0.39.0-kafka-3.5.1
                3.5.2=quay.io/strimzi/kafka:0.39.0-kafka-3.5.2
                3.6.0=quay.io/strimzi/kafka:0.39.0-kafka-3.6.0
                3.6.1=quay.io/strimzi/kafka:0.39.0-kafka-3.6.1
            - name: STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE
              value: quay.io/strimzi/operator:0.39.0
            - name: STRIMZI_DEFAULT_USER_OPERATOR_IMAGE
              value: quay.io/strimzi/operator:0.39.0
            - name: STRIMZI_DEFAULT_KAFKA_INIT_IMAGE
              value: quay.io/strimzi/operator:0.39.0
            - name: STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE
              value: quay.io/strimzi/kafka-bridge:0.27.0
            - name: STRIMZI_DEFAULT_KANIKO_EXECUTOR_IMAGE
              value: quay.io/strimzi/kaniko-executor:0.39.0
            - name: STRIMZI_DEFAULT_MAVEN_BUILDER
              value: quay.io/strimzi/maven-builder:0.39.0
            - name: STRIMZI_OPERATOR_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace

            - name: STRIMZI_FEATURE_GATES
              value: "+UnidirectionalTopicOperator"
            - name: JAVA_OPTS
              value: -Xms3g -Xmx3g
            - name: STRIMZI_ZOOKEEPER_SESSION_TIMEOUT_MS
              value: "900000"
            - name: KUBERNETES_CONNECTION_TIMEOUT
              value: "90000"
            - name: KUBERNETES_WEBSOCKET_TIMEOUT
              value: "50000"
            - name: KUBERNETES_REQUEST_TIMEOUT
              value: "90000"
            - name: STRIMZI_LEADER_ELECTION_ENABLED
              value: "true"
            - name: STRIMZI_LEADER_ELECTION_LEASE_NAME
              value: "strimzi-cluster-operator"
            - name: STRIMZI_LEADER_ELECTION_LEASE_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: STRIMZI_LEADER_ELECTION_IDENTITY
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          livenessProbe:
            httpGet:
              path: /healthy
              port: http
            initialDelaySeconds: 10
            periodSeconds: 30
          readinessProbe:
            httpGet:
              path: /ready
              port: http
            initialDelaySeconds: 10
            periodSeconds: 30
          resources:
            limits:
              cpu: 3
              memory: 6Gi
            requests:
              cpu: 1
              memory: 5Gi
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: eks.amazonaws.com/nodegroup
                operator: In
                values:
                - Kafka
      tolerations:
        - effect: NoSchedule
          key: kafka
          operator: Equal
          value: "true"
        - effect: NoExecute
          key: kafka
          operator: Equal
          value: "true"
       

[sachintandon-nexla]

@scholzj Please let me know if any more details are required.

[scholzj]

Well, probably the Kafka logs. The error says that the operator cannot talk with the Kafka node. So the issue will be in the Kafka node, or in some networking etc. The Kafka CR might be useful as well, but it might not tell us much either.