Issue exposing strimzi using Openshift route

[antonioproietti]

Hi all,
I'm trying to configure external access for strimzi-kafka on Openshift (internal access has always worked well), according to this
article on the strimzi blog for the route option. I read also similar previous github threads like this:
github.com//issues/6041

Mutual TLS authentication is not a must for us, so I tried to get things working initially without mutual authentication, but with no success.

Our environment: OCP version 3.11, Strimzi 0.17.0. Openshift nodes are on AWS cloud, so there is also a load balancer managing the *.apps.ocp.cluster-domain certificate.
For testing the reachability I was using the Offset Explorer 2.3 client.

My kafka configuration is like this:

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
......
spec:
....
  kafka:
    config:
      log.message.format.version: '2.4'
      message.max.bytes: 20971520
      offsets.topic.replication.factor: 1
      replica.fetch.max.bytes: 20971520
      transaction.state.log.min.isr: 1
      transaction.state.log.replication.factor: 1
    listeners:
      external:
        type: route
      plain: {}
      tls: {}
    replicas: 1
    resources:
    .......

According to the old listener sintax, TLS is enabled by default, so I saved the openshift ca certificate file:
oc extract secret/my-cluster-cluster-ca-cert --keys=ca.crt --to=- > ca.crt
And imported it in a java keystore (of type .jks)

I also create a "dedicated service for each of the brokers" (only one broker in this case) as suggested on the blog post.

Because between client and the OCP route there is the AWS load balancer with *.apps.ocp.cluster-domain wildcard certificate, I obtained a TLS handshake error. After adding also the AWS root certificate in the truststore, I was able to made a successful TLS handshake, but now there is a timeout error connecting to broker. From client log I can see:

24/Nov/2022 10:13:52.368 DEBUG org.apache.kafka.common.metrics.Metrics - Added sensor with name node--1.bytes-received
24/Nov/2022 10:13:52.368 DEBUG org.apache.kafka.common.metrics.Metrics - Added sensor with name node--1.latency
24/Nov/2022 10:13:52.368 DEBUG org.apache.kafka.common.network.Selector - [AdminClient clientId=adminclient-2] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -1
24/Nov/2022 10:13:52.399 DEBUG org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-2] Completed connection to node -1. Fetching API versions.
24/Nov/2022 10:13:52.556 DEBUG org.apache.kafka.common.network.SslTransportLayer - [SslTransportLayer channelId=-1 key=sun.nio.ch.SelectionKeyImpl@2a8e55d2] SSL handshake completed successfully with peerHost 'my-kafka-cluster-kafka-bootstrap-strimzi-test.apps.ocp.XXXXXX' peerPort 443 peerPrincipal 'CN=*.apps.ocp.XXXXXXX' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256'
24/Nov/2022 10:13:52.556 DEBUG org.apache.kafka.common.network.Selector - [AdminClient clientId=adminclient-2] Successfully authenticated with my-kafka-cluster-kafka-bootstrap-strimzi-XXXXXXXXXXXXX/18.198.XXX.XXX
24/Nov/2022 10:13:52.556 DEBUG org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-2] Initiating API versions fetch from node -1.
24/Nov/2022 10:13:52.665 ERROR org.apache.kafka.common.utils.KafkaThread - Uncaught exception in thread 'kafka-admin-client-thread | adminclient-2':
java.lang.OutOfMemoryError: Java heap space
	at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
	at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
	at org.apache.kafka.common.memory.MemoryPool$1.tryAllocate(MemoryPool.java:30)
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:112)
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:424)
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:385)
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:651)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:572)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:483)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:540)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1196)
	at java.lang.Thread.run(Thread.java:748)
24/Nov/2022 10:14:22.322 ERROR com.kafkatool.model.KafkaMapper - Timeout connecting to broker(s), bootstrap.servers=my-kafka-cluster-kafka-bootstrap-strimzi-test.apps.ocp.XXXXXXXXXXXXXX:443
java.util.concurrent.TimeoutException
	at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:108)
	at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:272)
	at com.kafkatool.model.KafkaMapper.getBrokersKafka(KafkaMapper.java:366)
	at com.kafkatool.model.KafkaMapper.getBrokers(KafkaMapper.java:350)
	at com.kafkatool.model.ServerConnection.getBrokers(ServerConnection.java:227)
	at com.kafkatool.model.ServerConnection.connectInt(ServerConnection.java:386)
	at com.kafkatool.model.ServerConnection.connect(ServerConnection.java:367)
	at com.kafkatool.common.AsyncServerConnector.run(AsyncServerConnector.java:43)

I also tried with another kafka client (Kadeck by Xeoteck, same boostrap url, same truststore configured) obtaining this error:

Could not complete ListTopics action: connection could not be established. TimeoutException: The AdminClient thread has exited. Call: listTopics

I missed something on the broker side? Checked the services and routes (for both bootstrap and broker): the endpoint is present and binded to target port 9094 as expected for external access...

Any suggestion is welcome, thanks

[scholzj]

You do not share any full configs of your broker, of the clients, full logs etc. So it is hard to comment. Some things I noticed:

there is also a load balancer managing the *.apps.ocp.cluster-domain certificate

That is not how it works. To get Kafka exposed using OpenShift routes, it needs ot use TLS passthrough so that the connection gets directly into the broker. So if you have some special setup where the loadbalancer interferes with the connection and does things such as TLS termination, then it will not work.

I also create a "dedicated service for each of the brokers" (only one broker in this case) as suggested on the blog post.

I'm not sure what do you try to follow. But you are not expected to create any services. The external listener in your configuration should create all the services and routes.

Because between client and the OCP route there is the AWS load balancer with *.apps.ocp.cluster-domain wildcard certificate, I obtained a TLS handshake error.

As said above ... there should be no certificate between the client and the route. If it interferes with TLS, it will not work.

I also tried with another kafka client (Kadeck by Xeoteck, same boostrap url, same truststore configured) obtaining this error:

Best is to download the Kafka binaries and use the kafka-console-consumer.sh and kafka-console-producer.sh scripts. Those are well known clients with well known configuration. So you can easily share everything and other will understand if you have it configured correctly.

[antonioproietti]

Hi and thanks for your quick reply.
About this point:

I also create a "dedicated service for each of the brokers" (only one broker in this case) as suggested on the blog post.
I'm not sure what do you try to follow. But you are not expected to create any services. The external listener in your configuration should create all the services and routes.

I confirm that services and routes was automatically made by the listener configuration, but in the article https://strimzi.io/blog/2019/04/30/accessing-kafka-part-3/, as explained by you, this service need to be associated with the statefulset.kubernetes.io/pod-name. It was just to say that those services/routes were exactly as expected from a standard setup.

Thank again for you help.