Operators throwing "Queued close did not succeed" errors

[ngc4579]

Hi,

we are running Strimzi Kafka Operator 0.39.0 in a hosted K8s environment (1.27.6). Most operator pods (Kafka Operator, Topic Operator, etc) are throwing regular errors like the following:

2024-02-01 08:34:42 WARN  JdkWebSocketImpl:129 - Queued close did not succeed
java.lang.IllegalArgumentException: statusCode
	at jdk.internal.net.http.websocket.WebSocketImpl.sendClose(WebSocketImpl.java:309) ~[java.net.http:?]
	at io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl.sendClose(JdkWebSocketImpl.java:126) ~[io.fabric8.kubernetes-httpclient-jdk-6.9.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onClose(WatcherWebSocketListener.java:67) ~[io.fabric8.kubernetes-client-6.9.2.jar:?]
	at io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl.onClose(JdkWebSocketImpl.java:84) ~[io.fabric8.kubernetes-httpclient-jdk-6.9.2.jar:?]
	at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.processClose(WebSocketImpl.java:528) ~[java.net.http:?]
	at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.run(WebSocketImpl.java:459) ~[java.net.http:?]
	at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:149) ~[java.net.http:?]
	at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230) ~[java.net.http:?]
	at jdk.internal.net.http.HttpClientImpl$DelegatingExecutor.execute(HttpClientImpl.java:157) ~[java.net.http:?]
	at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:305) ~[java.net.http:?]
	at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:274) ~[java.net.http:?]
	at jdk.internal.net.http.websocket.WebSocketImpl.trySetState(WebSocketImpl.java:845) ~[java.net.http:?]
	at jdk.internal.net.http.websocket.WebSocketImpl.signalClose(WebSocketImpl.java:771) ~[java.net.http:?]
	at jdk.internal.net.http.websocket.WebSocketImpl$SignallingMessageConsumer.onComplete(WebSocketImpl.java:828) ~[java.net.http:?]
	at jdk.internal.net.http.websocket.TransportImpl$ReceiveTask.run(TransportImpl.java:709) ~[java.net.http:?]
	at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:149) ~[java.net.http:?]
	at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230) ~[java.net.http:?]
	at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:303) ~[java.net.http:?]
	at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:256) ~[java.net.http:?]
	at jdk.internal.net.http.websocket.TransportImpl$ReadEvent.handle(TransportImpl.java:766) ~[java.net.http:?]
	at jdk.internal.net.http.RawChannelTube$ReadSubscriber.checkEvents(RawChannelTube.java:174) ~[java.net.http:?]
	at jdk.internal.net.http.RawChannelTube$ReadSubscriber.onComplete(RawChannelTube.java:223) ~[java.net.http:?]
	at jdk.internal.net.http.common.SSLTube$DelegateWrapper.onComplete(SSLTube.java:276) ~[java.net.http:?]
	at jdk.internal.net.http.common.SSLTube$SSLSubscriberWrapper.complete(SSLTube.java:440) ~[java.net.http:?]
	at jdk.internal.net.http.common.SSLTube$SSLSubscriberWrapper.onComplete(SSLTube.java:541) ~[java.net.http:?]
	at jdk.internal.net.http.common.SubscriberWrapper.checkCompletion(SubscriberWrapper.java:472) ~[java.net.http:?]
	at jdk.internal.net.http.common.SubscriberWrapper$DownstreamPusher.run1(SubscriberWrapper.java:334) ~[java.net.http:?]
	at jdk.internal.net.http.common.SubscriberWrapper$DownstreamPusher.run(SubscriberWrapper.java:259) ~[java.net.http:?]
	at jdk.internal.net.http.common.SequentialScheduler$LockingRestartableTask.run(SequentialScheduler.java:205) ~[java.net.http:?]
	at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:149) ~[java.net.http:?]
	at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230) ~[java.net.http:?]
	at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:303) ~[java.net.http:?]
	at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:256) ~[java.net.http:?]
	at jdk.internal.net.http.common.SubscriberWrapper.outgoing(SubscriberWrapper.java:232) ~[java.net.http:?]
	at jdk.internal.net.http.common.SSLFlowDelegate$Reader.processData(SSLFlowDelegate.java:513) ~[java.net.http:?]
	at jdk.internal.net.http.common.SSLFlowDelegate$Reader$ReaderDownstreamPusher.run(SSLFlowDelegate.java:268) ~[java.net.http:?]
	at jdk.internal.net.http.common.SequentialScheduler$LockingRestartableTask.run(SequentialScheduler.java:205) ~[java.net.http:?]
	at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:149) ~[java.net.http:?]
	at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230) ~[java.net.http:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-02-01 08:34:42 WARN  JdkWebSocketImpl:142 - Aborting WebSocket due to a write error or failure with sendClose

The location indicates something going wrong inside the fabric8 Kubernetes client (I might file a discussion or bug report there as well). Does anyone have a clue what causes this, or where to start looking for the root cause? As far as I can tell all operators are working normally apart from those frequent error messages (in fact it repeats every minute)...

[scholzj]

It is not even clear from which component it is. But it looks like the connection to your Kubernetes APi server is breaking. This could be because of networking issues, some software instability, surely it can have something to do with resources as well etc. So almost anything.

[scholzj]

I guess that could definitely be a problem. But I have no idea what would cause it. Sometimes it is stupid things such as a pod running on a broken worker node. Othertimes it could be complicated :-(.

[ngc4579]

Which fabric8 Kubernetes client version does the Strimzi Kafka Operator 0.39.0 use? Strimzi Kafka Operator 0.39.0 uses fabric8 Kubernetes client 6.9.2, right? Guess that'll be of interest when starting a discussion on their GitHub...

[scholzj]

TBH, you would need to check in the container image. Exec into the operator image and check in lib cirectory. I thought it was 6.10, but I might be wrong.

I'm not aware of anyone else having this kind of issue. So I doubt it is a Fabric8 issue. But they might help you to debug the issue or have some pointers to look into. Your exact environment might be important as well. As some Kubernetes distributions do some things in a special/weird ways sometimes.

[ngc4579]

[strimzi@strimzi-cluster-operator-6d964b677c-qww7l lib]$ ls | grep fabric8
io.fabric8.kubernetes-client-6.9.2.jar
io.fabric8.kubernetes-client-api-6.9.2.jar

🙃

[scholzj]

Right, so looks like 6.10 was only after 0.39 ... time goes by quickly, it feels like ages since I bumped it to 6.10 ... but I guess the 0.39 release is now over a month old as well, so maybe that is why it feels like it :-o.

[Totalus]

Hey !
I am observing the same issue on my cluster since I made few udpates to kafka and strimzi and K8S itself.

It seems like Strimzi 0.39 was first running properly at first, and only started to log that error shortly after I launch a K8S cluster version upgrade. I would expect this kind of error during the upgrade process, but it did not stop after the upgrade and keeps logging errors since then. I get that error on average each 1.3 seconds, which is a pretty high rate.

Versions

• K8S 1.26.11
• Strimzi 0.39

@ngc4579 Did you also upgrade your cluster or did any cluster-related operation when those errors start to pop ?

[ngc4579]

@Totalus Thanks for your feedback. No, I did not perform any cluster-related operation. The error appeared right from the beginning when I first started to play around with the Strimzi Kafka Operator, even with a completely basic (i.e. without specific configuration) Helm install.

The operators themselves seem to work normally, apart from that high error log frequency - I've deployed a 3-node Kafka cluster with connectors that has been running without issues so far.