Skip to content

Commit

Permalink
docs(topic operator): describes the tuning for topic operator batchin…
Browse files Browse the repository at this point in the history
…g mechanism (#9314)

Signed-off-by: prmellor <[email protected]>
  • Loading branch information
PaulRMellor authored Nov 8, 2023
1 parent 9a83ce1 commit 4d603df
Show file tree
Hide file tree
Showing 3 changed files with 25 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -28,3 +28,5 @@ include::../../modules/operators/con-deleting-managed-topics.adoc[leveloffset=+1
include::../../modules/operators/con-changing-topic-operator-mode.adoc[leveloffset=+1]
//removing finalizers from topics
include::../../modules/operators/con-removing-topic-finalizers.adoc[leveloffset=+1]
//tuning request batches
include::../../modules/operators/con-tuning-topic-request-batches.adoc[leveloffset=+1]
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
// Module included in the following assemblies:
//
// assembly-using-the-topic-operator.adoc

[id='con-tuning-topic-request-batches-{context}']
= (Preview) Tuning request batches for topic operations

In unidirectional mode, the Topic Operator uses the request batching capabilities of the Kafka Admin API for operations on topic resources.
You can fine-tune the batching mechanism using the following operator configuration properties:

* `STRIMZI_MAX_QUEUE_SIZE` to set the maximum size of the topic event queue.
The default value is 1024.
* `STRIMZI_MAX_BATCH_SIZE` to set the maximum number of topic events allowed in a single batch.
The default value is 100.
* `MAX_BATCH_LINGER_MS` to specify the maximum time to wait for a batch to accumulate items before processing.
The default is 100 milliseconds.

If the maximum size of the request batching queue is exceeded, the Topic Operator shuts down and is restarted.
To prevent frequent restarts, consider adjusting the `STRIMZI_MAX_QUEUE_SIZE` property to accommodate the typical load.
5 changes: 4 additions & 1 deletion documentation/modules/operators/ref-operator-topic.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -30,4 +30,7 @@ In unidirectional mode, the Topic Operator operates as follows:

If a topic is created, deleted, or modified directly within the Kafka cluster, without the presence of a corresponding `KafkaTopic` resource, the Topic Operator does not manage that topic.
The Topic Operator will only manage Kafka topics associated with `KafkaTopic` resources and does not interfere with topics managed independently within the Kafka cluster.
If a `KafkaTopic` does exist for a Kafka topic, any configuration changes made outside the resource are reverted.
If a `KafkaTopic` does exist for a Kafka topic, any configuration changes made outside the resource are reverted.

The Topic Operator can detect cases where where multiple `KafkaTopic` resources are attempting to manage a Kafka topic using the same `.spec.topicName`.
Only the oldest resource is reconciled, while the other resources fail with a resource conflict error.

0 comments on commit 4d603df

Please sign in to comment.