Should I create too many topics in Kafka?

[clyang82]

My requirement is as a cluster admin, I can isolate my cluster data so that the user can only access the data/message who has permission to do. In order to meet this requirement, the intuitional solution is to leverage the existing feature to define the kafka user to access the certain topics with pre-defined ACLs.

If I want to manage many clusters, for example 10K managed clusters, I need to create 10K kafkausers and 10K topics at least (one user and one topic per cluster). I searched if Kafka can support many topics. From confluent document

While there is no set limit to the number of topics that can exist in a Kafka cluster, currently Kafka can handle hundreds of thousands of topics, depending on the number of partitions in each.

It seems I can do in this way. So I use the admin API to create 10K users and 10K topics with ACLs ( It has problem to create 10K kafkaUser and kafkaTopic CR), it works. But I guess it may have the performance issues if the action triggers the re-balance. Does this solution make sense for you to support large scale with ACLs? any suggestions for large scale environment with authorization? Thanks in advance.

[scholzj]

If I want to manage many clusters, for example 10K managed clusters, I need to create 10K kafkausers and 10K topics at least (one user and one topic per cluster).

I think it depends on what exactly are you asking for:

• 10k Kafka clusters each with 10k topics and users each? How would that match against Kubernetes clusters? Would each be in its own Kubernetes clusters? Or will they share the Kubernetes cluster? If they would share the same Kube cluster, I would expect that you would run into some resource limits with Kubernetes itself.
• 10k Kafka clusters with 1 topic and 1 user each? Again, it would depend on how that would match the Kubernetes clusters. If it would be one Kubernetes cluster, 10k custom resources of each might be fine. But you will likely run into limitations of how many Kafka clusters the Strimzi Cluster Operator can handle. Also, this will cost you a huge amount of resources just to run the Kafka clusters. So I'm not sure what the purpose of this really is, but it does not sound like the right way to do it.

Of course, using the User and Topic operators is completely optional. So you can also manage things directly using Kafka Admin API if you want.

[yanmxa]

I think before talking about HA with Kafka, we need to understand why you picked Kafka and if there are good reasons for that.

Thanks @ppatierno! It's a good question! IMHO the reason to consider using Kafka here is to take advantage of its authentication and authorization mechanism. Because Kafka's ACL/permission management is based on the topic(if I understand correctly), each cluster will need an exclusive topic for permission control. And a lot of clusters will need to create a lot of topics(10k)...

A simple scenario: clusterA cannot consume the messages for clusterB. Also, clusterA cannot pretend to be clusterB to send messages to server-side by the transport.

Regarding rebalancing ... what are referring to? A rebalancing of multiple partitions to different consumers happens when consumers join/left a consumer group but in your case AFAICS you have one consumer (agent) per topic with just one partition so no rebalance. Also if you are talking about load rebalancing via Cruise Control, it's something you trigger when/if needed, it's not automatic.

The rebalance @clyang82 referred to is on the server side. where the server subscribes to a pattern topic like ^status.*. Then the operator finds a new cluster named clusterX joined and creates the associated topic status.clusterX. Is the rebalance triggered on the server side? I think it happens, cause the new topic/partition will join the server-side consumer group.

[ppatierno]

Due to AMQ Broker cannot support authn and authz with mTLS ...

Is it really true? It sounds really weird to me that upstream Artemis (so AMQ Broker) doesn't have such a feature. @brusdev any thoughts as maintainer on that project?

I am asking because I see it as a scenario for a messaging broker whatever is the protocol. So maybe not using MQTT but i.e. AMQP instead.

[brusdev]

@clyang82 @ppatierno I confirm AMQ Broker support authn and authz with mTLS, you can use the CertificateLoginModule, for further detail see https://activemq.apache.org/components/artemis/documentation/latest/security.html#dual-authentication

[ppatierno]

Thanks @ppatierno! It's a good question! IMHO the reason to consider using Kafka here is to take advantage of its authentication and authorization mechanism. Because Kafka's ACL/permission management is based on the topic(if I understand correctly), each cluster will need an exclusive topic for permission control. And a lot of clusters will need to create a lot of topics(10k)...
A simple scenario: clusterA cannot consume the messages for clusterB. Also, clusterA cannot pretend to be clusterB to send messages to server-side by the transport.

I think I have got the scenario, so for this reason I am wondering why a messaging broker doesn't work for you. Having maybe one queue per maestro agent? With authn/authz on it? Something where the message flows, is consumed and then deleted because you save everything in a database. While Kafka is providing more features which you don't use.

Is the rebalance triggered on the server side? I think it happens, cause the new topic/partition will join the server-side consumer group.

Well technically it's not a topic partition joining a consumer group. It's a consumer joining the group. So the server will be the only one member of such a group subscribing to a pattern. Said that I have to verify if a new topic matches the pattern it will trigger a rebalance where anyway the additional partition will be assigned still to the same single consumer.

[see-quick]

I think there are several aspects you need to understand when it comes to the performance. The initial creations of Xk topics is one part of it. But you should also understand for example when you have 10k existing topics that exist and are ready and you create the 10001st -> will it create in few seconds? Or will it create the new topic in 2 hours? The same applies to when you change some configuration.

Just to add my one bit to the performance side...I did not try/test the creation of the 10001st KafkaTopic but I tried to update 10 KafkaTopics configs (already created) and it took around 3s to complete (for all 10 topics to change a config).

[clyang82]

@clyang82 @ppatierno I confirm AMQ Broker support authn and authz with mTLS, you can use the CertificateLoginModule, for further detail see https://activemq.apache.org/components/artemis/documentation/latest/security.html#dual-authentication

Thanks @brusdev for your information. Here is an original issue logged - https://github.com/artemiscloud/activemq-artemis-operator/issues/724. Could you give us the advice? Thanks.

[brusdev]

@clyang82 I replied in https://github.com/artemiscloud/activemq-artemis-operator/issues/724. It is already resolved and will be closed when the documentation that explains how to set up the dual authentication by using the ActiveMQArtemis CR is added.

[clyang82]

That is a good news. Thanks @brusdev will try it soon.