No space left on device from strimzi-cluster-operator

[elvis-cai]

HI team

we found operator can not reconcille kafka due to, may be related to #7536,
however this time tmpfs 1.0M 1016K 8.0K 100% /tmp is full and we can see a lot of tls files

30
2023-01-10 00:24:33 WARN  AbstractOperator:481 - Reconciliation #51355(timer) Kafka(dev/kafka): Failed to reconcile
29
java.lang.RuntimeException: java.io.IOException: No space left on device
28
	at io.strimzi.operator.cluster.model.Ca.generateCaKeyAndCert(Ca.java:941) ~[io.strimzi.operator-common-0.25.0.jar:0.25.0]
27
	at io.strimzi.operator.cluster.model.Ca.createRenewOrReplace(Ca.java:530) ~[io.strimzi.operator-common-0.25.0.jar:0.25.0]
26
	at io.strimzi.operator.cluster.operator.assembly.KafkaAssemblyOperator$ReconciliationState.lambda$reconcileCas$6(KafkaAssemblyOperator.java:637) ~[io.strimzi.cluster-operator-0.25.0.jar:0.25.0]
25
	at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:160) ~[io.vertx.vertx-core-4.1.2.jar:4.1.2]
24
	at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:96) ~[io.vertx.vertx-core-4.1.2.jar:4.1.2]
23
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:158) ~[io.vertx.vertx-core-4.1.2.jar:4.1.2]
22
	at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-4.1.2.jar:4.1.2]
21
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
20
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
19
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.66.Final.jar:4.1.66.Final]
18
	at java.lang.Thread.run(Thread.java:829) [?:?]
17
Caused by: java.io.IOException: No space left on device
16
	at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:?]
15
	at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:62) ~[?:?]
14
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113) ~[?:?]
13
	at sun.nio.ch.IOUtil.write(IOUtil.java:79) ~[?:?]
12
	at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:280) ~[?:?]
11
	at java.nio.channels.Channels.writeFullyImpl(Channels.java:74) ~[?:?]
10
	at java.nio.channels.Channels.writeFully(Channels.java:97) ~[?:?]
9
	at java.nio.channels.Channels$1.write(Channels.java:172) ~[?:?]
8
	at java.io.InputStream.transferTo(InputStream.java:705) ~[?:?]
7
	at java.nio.file.Files.copy(Files.java:3078) ~[?:?]
6
	at io.strimzi.certs.OpenSslCertManager.createDefaultConfig(OpenSslCertManager.java:103) ~[io.strimzi.certificate-manager-0.25.0.jar:0.25.0]
5
	at io.strimzi.certs.OpenSslCertManager.generateCaCert(OpenSslCertManager.java:269) ~[io.strimzi.certificate-manager-0.25.0.jar:0.25.0]
4
	at io.strimzi.certs.OpenSslCertManager.generateRootCaCert(OpenSslCertManager.java:166) ~[io.strimzi.certificate-manager-0.25.0.jar:0.25.0]
3
	at io.strimzi.certs.OpenSslCertManager.generateSelfSignedCert(OpenSslCertManager.java:145) ~[io.strimzi.certificate-manager-0.25.0.jar:0.25.0]
2
	at io.strimzi.operator.cluster.model.Ca.generateCaKeyAndCert(Ca.java:919) ~[io.strimzi.operator-common-0.25.0.jar:0.25.0]
1
	... 10 more

-rw-r--r-- 1 strimzi root    0 Jan  9 13:46 tls9667619142783164337broker-cert
-rw-r--r-- 1 strimzi root    0 Jan  9 06:42 tls9673619171562178936broker-csr
-rw-r--r-- 1 strimzi root    0 Jan  9 17:52 tls9679991702647893836broker-key
-rw-r--r-- 1 strimzi root    0 Jan  9 12:46 tls9680904466919732608broker-p12
-rw-r--r-- 1 strimzi root    0 Jan  9 14:58 tls9690279095811396920broker-csr
-rw-r--r-- 1 strimzi root    0 Jan  9 01:42 tls9690732429476349875broker-key
-rw-r--r-- 1 strimzi root    0 Jan  8 23:44 tls969460180759168131broker-key
-rw-r--r-- 1 strimzi root    0 Jan  9 06:00 tls9695662275411430716broker-key
-rw-r--r-- 1 strimzi root    0 Jan  9 06:50 tls970435335614512759broker-p12
-rw-r--r-- 1 strimzi root    0 Jan  9 15:28 tls9713287805745439812broker-cert
-rw-r--r-- 1 strimzi root    0 Jan  9 17:14 tls9717430568985024711broker-key
-rw-r--r-- 1 strimzi root    0 Jan  9 21:52 tls9721653153272600631broker-cert
-rw-r--r-- 1 strimzi root    0 Jan  9 00:26 tls9725556093409979345broker-cert
-rw-r--r-- 1 strimzi root    0 Jan  9 12:42 tls9727578212812702948broker-cert
-rw-r--r-- 1 strimzi root    0 Jan  9 10:22 tls9731259492670883579broker-cert
-rw-r--r-- 1 strimzi root    0 Jan  8 23:20 tls9737132791836050605broker-csr
-rw-r--r-- 1 strimzi root    0 Jan  9 15:42 tls9741792178873518592broker-p12
-rw-r--r-- 1 strimzi root    0 Jan  9 16:22 tls9743697210521185109broker-csr
-rw-r--r-- 1 strimzi root    0 Jan  9 16:42 tls9752144539724435848broker-cert
-rw-r--r-- 1 strimzi root    0 Jan  9 11:00 tls975248051640048499broker-key
-rw-r--r-- 1 strimzi root    0 Jan  9 10:40 tls9755967677975098449broker-p12
-rw-r--r-- 1 strimzi root    0 Jan  9 11:46 tls9760863525524060868broker-key
-rw-r--r-- 1 strimzi root    0 Jan  9 16:26 tls9761134157429246684broker-cert
-rw-r--r-- 1 strimzi root    0 Jan  9 19:30 tls9762227308818934035broker-p12
-rw-r--r-- 1 strimzi root    0 Jan  9 19:58 tls9766151426001515345broker-csr
-rw-r--r-- 1 strimzi root    0 Jan  9 13:48 tls9776817868098559680broker-key
-rw-r--r-- 1 strimzi root    0 Jan  9 20:12 tls9778280650285034785broker-csr
-rw-r--r-- 1 strimzi root    0 Jan  9 18:10 tls9781971326157026544broker-csr
-rw-r--r-- 1 strimzi root    0 Jan  9 10:12 tls9787967411310176890broker-cert
-rw-r--r-- 1 strimzi root    0 Jan  9 13:20 tls9788034194212248702broker-key
-rw-r--r-- 1 strimzi root    0 Jan  9 12:24 tls979675885759850529broker-key
-rw-r--r-- 1 strimzi root    0 Jan  9 10:04 tls9797321339613767975broker-csr
-rw-r--r-- 1 strimzi root    0 Jan  9 20:18 tls9803024048472628281broker-csr
-rw-r--r-- 1 strimzi root    0 Jan  9 17:48 tls9805901901056940579broker-csr
-rw-r--r-- 1 strimzi root    0 Jan  9 16:18 tls9810124174065546074broker-cert
-rw-r--r-- 1 strimzi root    0 Jan  9 03:00 tls9811191001454327861broker-key
-rw-r--r-- 1 strimzi root    0 Jan  9 10:34 tls9813232424711743249broker-p12
-rw-r--r-- 1 strimzi root    0 Jan  9 12:22 tls9820343880018648152broker-key
-rw-r--r-- 1 strimzi root    0 Jan  9 03:58 tls9822175115486917306broker-p12
-rw-r--r-- 1 strimzi root    0 Jan  9 07:06 tls982754315334941494broker-key
-rw-r--r-- 1 strimzi root    0 Jan  8 22:56 tls9828411093645025912broker-key
-rw-r--r-- 1 strimzi root    0 Jan  9 08:10 tls9834521537641434582broker-key
-rw-r--r-- 1 strimzi root    0 Jan  9 20:46 tls9837166948111913224broker-cert
-rw-r--r-- 1 strimzi root    0 Jan  9 02:14 tls9849239795405999672broker-csr
-rw-r--r-- 1 strimzi root    0 Jan  9 08:28 tls9853901697921432929broker-p12
-rw-r--r-- 1 strimzi root    0 Jan  9 19:10 tls9854501448291356942broker-cert
-rw-r--r-- 1 strimzi root    0 Jan  9 08:26 tls9861465717396870465broker-key
-rw-r--r-- 1 strimzi root    0 Jan  9 14:24 tls9882870904027166447broker-cert
-rw-r--r-- 1 strimzi root    0 Jan  9 07:10 tls9885962811934474814broker-p12
-rw-r--r-- 1 strimzi root    0 Jan  9 19:02 tls988843063929793365broker-cert
-rw-r--r-- 1 strimzi root    0 Jan  8 23:38 tls9901190336161547376broker-p12
-rw-r--r-- 1 strimzi root    0 Jan  9 13:04 tls9901943513995671487broker-cert
-rw-r--r-- 1 strimzi root    0 Jan  9 03:00 tls9909952038225424855broker-p12
-rw-r--r-- 1 strimzi root    0 Jan  9 20:32 tls9911816215471704744broker-key
-rw-r--r-- 1 strimzi root    0 Jan  9 16:54 tls9912556487960500324broker-p12
-rw-r--r-- 1 strimzi root    0 Jan  8 21:55 tls9913800447750111619broker-cert
-rw-r--r-- 1 strimzi root    0 Jan  8 21:55 tls9916331684255496460broker-csr
-rw-r--r-- 1 strimzi root    0 Jan  9 15:50 tls9923541648522040343broker-cert
-rw-r--r-- 1 strimzi root    0 Jan  9 11:28 tls9936902767579899656broker-csr
-rw-r--r-- 1 strimzi root    0 Jan  9 13:22 tls9941215900883174822broker-csr
-rw-r--r-- 1 strimzi root    0 Jan  9 07:32 tls9962740152292663641broker-key
-rw-r--r-- 1 strimzi root    0 Jan  9 01:46 tls9968822428219763623broker-cert
-rw-r--r-- 1 strimzi root    0 Jan  9 03:22 tls9977779898641918073broker-cert
-rw-r--r-- 1 strimzi root    0 Jan  9 09:26 tls9977864140577134270broker-csr
-rw-r--r-- 1 strimzi root    0 Jan  9 12:46 tls9987119317281745243broker-key
-rw-r--r-- 1 strimzi root    0 Jan  9 00:50 tls9990268632911485327broker-cert
-rw-r--r-- 1 strimzi root    0 Jan  9 08:06 tls9992968486875831033broker-key

Here's the full tmp list https://gist.github.com/elvis-cai/c0f0852677c118e0ad4e8ae2f3b189c9

[elvis-cai]

we are using quay.io/strimzi/operator:0.25.0, so probably we need upgrade our operator due to this fix? #5836

[scholzj]

Yeah, there were certainly some fixes to how the files are managed. I do not have any reports from latest Strimzi versions about this issue. So I think upgrading would be the best way to proceed.

[elvis-cai]

understood, thanks @scholzj