Strimzi operator crashed on reconciliation

[nevlkv]

Strimzi operator 0.27.1 + Drain cleaner (kafka + zookeeper)

When k8 node became drained Drain cleaner annotate broker and zookeper pods strimzi.io/manual-rolling-update: true

Strimzi operator start manual reboot pods. Zookeper pod successfully rebooted and stay pending, strimzi operator keep failing with

2022-12-14 07:53:30 ERROR Util:149 - Reconciliation #3945(timer) Kafka(io-kafka/common-n2): Exceeded timeout of 300000ms while waiting for Pods resource common-n2-zookeeper-3 in namespace io-kafka to be ready
2022-12-14 07:53:30 ERROR AbstractOperator:247 - Reconciliation #3945(timer) Kafka(io-kafka/common-n2): createOrUpdate failed
io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for Pods resource common-n2-zookeeper-3 in namespace io-kafka to be ready
	at io.strimzi.operator.common.Util$1.lambda$handle$1(Util.java:150) ~[io.strimzi.operator-common-0.27.1.jar:0.27.1]
	at io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
	at io.vertx.core.impl.future.FutureBase.lambda$emitFailure$1(FutureBase.java:69) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) [io.netty.netty-transport-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2022-12-14 07:53:30 WARN  AbstractOperator:532 - Reconciliation #3945(timer) Kafka(io-kafka/common-n2): Failed to reconcile
io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for Pods resource common-n2-zookeeper-3 in namespace io-kafka to be ready
	at io.strimzi.operator.common.Util$1.lambda$handle$1(Util.java:150) ~[io.strimzi.operator-common-0.27.1.jar:0.27.1]
	at io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
	at io.vertx.core.impl.future.FutureBase.lambda$emitFailure$1(FutureBase.java:69) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) [io.netty.netty-transport-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2022-12-14 07:54:30 INFO  ClusterOperator:121 - Triggering periodic reconciliation for namespace *

and broker pod never rebooted with strimzi.io/manual-rolling-update: true

Name:                 common-n2-kafka-0
Namespace:            io-kafka
Priority:             1000100
Priority Class Name:  topolvm
Node:                 name/ip
Start Time:           Tue, 22 Nov 2022 10:51:41 +0300
Labels:               app.kubernetes.io/instance=common-n2
                      app.kubernetes.io/managed-by=strimzi-cluster-operator
                      app.kubernetes.io/name=kafka
                      app.kubernetes.io/part-of=strimzi-common-n2
                      controller-revision-hash=common-n2-kafka-f866c589b
                      statefulset.kubernetes.io/pod-name=common-n2-kafka-0
                      strimzi.io/cluster=common-n2
                      strimzi.io/kind=Kafka
                      strimzi.io/name=common-n2-kafka
Annotations:          cni.projectcalico.org/containerID: d4959a46f854d4e9e1b051a146e726bb5a81062a3de9d5ae08d832f6df4070da
                      cni.projectcalico.org/podIP: IP/32
                      cni.projectcalico.org/podIPs: IP/32
                      kubernetes.io/psp: 050-ns-io-kafka
                      prometheus.io/port: 9404
                      prometheus.io/scrape: true
                      strimzi.io/broker-configuration-hash: 0ba1e590a67384ecc6eaa4d00801811da6fd5a55b331c5d5aed77572647cf50b793f6aa05696c993
                      strimzi.io/clients-ca-cert-generation: 1
                      strimzi.io/cluster-ca-cert-generation: 1
                      strimzi.io/generation: 17
                      strimzi.io/inter-broker-protocol-version: 2.8
                      strimzi.io/kafka-version: 2.8.0
                      strimzi.io/log-message-format-version: 2.8
                      strimzi.io/logging-appenders-hash: e893ac9f32f7958042d1c82477e3484b0f8ebad5
                      strimzi.io/manual-rolling-update: true
                      strimzi.io/storage: {"volumes":[{"type":"persistent-claim","size":"439Gi","class":"topolvm-ext4","id":0}],"type":"jbod"}
Status:               Running
IP:                   IP

[scholzj]

The operator intentionally doesn't continue to stop more pods if the pods are not recovering. Without this behaviour, it could happen that it would this way after some time kill your whole cluster by rolling always the next pod without the previous one recovering. So you need to look into why the ZooKeeper pod is pending and not running and getting ready in the first place.

[nevlkv]

zookeeper and kafka broker in one k8s node, pv located on drained node. Zookeper always stay pending, operator always crash

[scholzj]

Hmm, ok ... I get your scenario, it makes sense. But I unfortunately do not think the Drain Cleaner mechanism was really designed for it. The operator will today not roll the Kafka brokers while the ZooKeeper cluster is not completely ready. We would need to change it to move to Kafka if it at least has a quorum instead of waiting for ZooKeeper to be fully ready. If you want, I can open an enhancement issues to track it. But TBH, as we focus more on removing ZooKeeper these days, I'm not sure anyone will get to it.

[nevlkv]

For resolve my problem i disable operate with zookeeper by drain cleaner, you may describe in documentation about kafka @ zookeper podAntiAfinity