Upgrading Strimzi Kafka operator from 0.42.0 to 0.43.0 fails with 'Failed to parse default container image configuration for Kafka from environment variable'

[abregar]

I recently transitioned to using Strimzi for setting up Kafka in k8s, beginning with Strimzi version 0.42.0. I carefully consulted the documentation on installation and upgrade procedures. However, during a recent attempt to upgrade to version 0.43.0, I encountered an issue that I would like to clarify with the community.

The starting point is a fully operational cluster based on Strimzi v0.42.0, managed via Helm and ArgoCD. Below is a partial view of the Kafka manifest, showing that, according to the compatibility matrix, this should be the latest compatible version of Kafka and IBPV:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: nodeport-test
  namespace: kafka
spec:
  kafka:
    version: 3.7.1
    replicas: 3
    ...
    config:
      ...
      inter.broker.protocol.version: "3.7"

The issue arises immediately after upgrading the Operator, where it fails with the following error:

Exception in thread "main" io.strimzi.operator.common.InvalidConfigurationException: Failed to parse default container image configuration for Kafka from environment variable STRIMZI_KAFKA_IMAGES
        at io.strimzi.operator.cluster.ClusterOperatorConfig.parseKafkaVersions(ClusterOperatorConfig.java:367)
        at io.strimzi.operator.cluster.ClusterOperatorConfig.buildFromMap(ClusterOperatorConfig.java:286)
        at io.strimzi.operator.cluster.Main.main(Main.java:62)
Caused by: io.strimzi.operator.cluster.model.NoImageException: No image for version 3.6.1 in {3.6.0=quay/strimzi/kafka:0.42.0-kafka-3.6.0, 3.7.0=quay/strimzi/kafka:0.42.0-kafka-3.7.0, 3.7.1=quay/strimzi/kafka:0.42.0-kafka-3.7.1, 3.8.0=quay/strimzi/kafka:0.42.0-kafka-3.8.0}
        at io.strimzi.operator.cluster.model.KafkaVersion$Lookup.image(KafkaVersion.java:221)
        at io.strimzi.operator.cluster.model.KafkaVersion$Lookup.validateImages(KafkaVersion.java:257)
        at io.strimzi.operator.cluster.model.KafkaVersion$Lookup.validateKafkaImages(KafkaVersion.java:274)
        at io.strimzi.operator.cluster.ClusterOperatorConfig.parseKafkaVersions(ClusterOperatorConfig.java:353)

After browsing through the documentation and existing issues, I’ve gathered that Kafka versions are not meant to be configured or overridden manually but are strictly determined by the Operator's Helm templates.

So, I kindly ask for clarification on two points:

• Where is the version 3.6.x being sourced from?
• Why are the images still marked with the tag (suffix) 0.42.0?

[scholzj]

Did you checked how the Cluster Operator deployment looks like? This seems to me like it wasn't properly updated and still has some old images configured (or maybe has the new images configured but is actually still the old version). But hard to be more exact without full operator logs and possible the Operator Deployment / Pod YAMLs.

[abregar]

The deployment is quite straightforward, essentially a direct copy from the documentation, just some values more narrowed down, i.e.:

watchNamespaces: [kafka]
watchAnyNamespace: false

defaultImageRegistry: imagecache.host.net/quay
defaultImageRepository: strimzi
defaultImageTag: 0.42.0

Follow-up: After further investigation, it seems the issue may be related to how ArgoCD handles Helm deployments.

I tested installing the application directly with Helm, bypassing ArgoCD, and that method worked without issues. Additionally, upgrading from version 0.42.0 to 0.43.0 using Helm directly was successful. However, when upgrading ArgoCD-managed deployments, it appears that ArgoCD did not apply the necessary labels and annotations to the Helm releases correctly. As a result, I had to manually update the labels and annotations on the existing objects. After making these adjustments, the upgrade via Helm was successful, and the operator started without errors.

Before concluding this discussion, I plan to further consult ArgoCD's documentation, particularly regarding the nuances between install, upgrade, and sync. According to the ArgoCD documentation:

"Argo CD cannot distinguish between a first-time 'install' or an 'upgrade'—every operation is treated as a 'sync'. By default, apps with pre-install and pre-upgrade hooks will have those hooks executed during the sync process."

I have suspicion that this messes up the template running.