Kafka introduced tiered storage feature since 3.6. The feature is in early stage and it’s nice to support tiered storage configuration natively in Strimzi.
Supporting tiered storage in Strimzi would allow Strimzi users to enable tiered storage natively with Strimzi config. By allowing to config custom configuration of Kafka's RemoteStorageManager interface, Strimzi users would be able to easily adopt the tiered storage feature on Strimzi easily. More details about RemoteStorageManager can be found in KIP405.
For simplicity, the proposal doesn't add support to customize RemoteLogMetadataManager. If tiered storage is enabled, the default implementation class TopicBasedRemoteLogMetadataManager in Kafka will be used.
This proposal focuses on tiered storage support with a custom plugin brought in by the user. Other proposals in the future might add some built-in support for selected storage implementations such as S3. It should also be noted that, it’s on the end-user to ensure that their tiered storage setup works with Strimzi; the operator itself is strictly responsible for pushing down the necessary config to the broker pods.
Add a spec.kafka.tieredStorage field in the Kafka CR, which would support configuration for all tiered storage related configuration.
The tieredStorage config contains configuration for some selected configuration of tiered storage feature. Most importantly, the configuration for RemoteStorageManager is needed.
kafka:
tieredStorage:
type: custom
remoteStorageManager:
className: <ClassName>
classPath: <ClassPath>
config:
# A map with String keys and String values. Key fields will be automatically prefixed with `rsm.config.` and appended to Kafka broker config.
<key>: <value>
The RSM config only supports custom type. The config allows users to specify the RSM class information.
Because the RemoteStorageManager dependency is provided by a Kafka plugin, the implementation could vary case by case. It’s important to allow users to pass in additional configuration values within the supplied prefix range. The custom configuration can be supplied via spec.kafka.config with the predefined prefix.
For simplicity, the proposal doesn't add customization for RemoteLogMetadataManager. If tiered storage is enabled, the default implementation using TopicBasedRemoteLogMetadataManager will be used, and corresponding configuration will be appended.
The Strimzi Cluster Operator will automatically configure any required configurations for the RLMM such as the basic thread pool configs required to create the internal Kafka client and talk to the brokers.
Any additional RLMM configuration options such as performance tuning or security config options can be put in .spec.kafka.config directly using the rlmm.config. prefix.
Example can be found below.
The below config define a sample configuration for tiered storage setup, using a custom RSM type.
kafka:
tieredStorage:
type: custom
remoteStorageManager:
className: com.example.kafka.tiered.storage.s3.S3RemoteStorageManager
classPath: /opt/kafka/plugins/tiered-storage-s3/*
config:
# A map with String keys and String values. Key fields will be automatically prefixed with `rsm.config.` and appended to Kafka broker config.
storage.bucket.name: my-bucket
config:
...
rlmm.config.remote.log.metadata.topic.replication.factor: 1
The configuration above will get translated to the below Kafka broker config:
# RLMM configuration generated by Strimzi
remote.log.storage.system.enable: true
remote.log.metadata.manager.impl.prefix: rlmm.config.
remote.log.metadata.manager.class.name: org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager
remote.log.metadata.manager.listener.name=REPLICATION-9091
rlmm.config.remote.log.metadata.common.client.security.protocol=SSL
rlmm.config.remote.log.metadata.common.client.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
rlmm.config.remote.log.metadata.common.client.ssl.keystore.password=${CERTS_STORE_PASSWORD}
rlmm.config.remote.log.metadata.common.client.ssl.keystore.type=PKCS12
rlmm.config.remote.log.metadata.common.client.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
rlmm.config.remote.log.metadata.common.client.ssl.keystore.password=${CERTS_STORE_PASSWORD}
rlmm.config.remote.log.metadata.common.client.ssl.truststore.type=PKCS12
#RSM configs set by the operator and by the user
remote.log.storage.manager.class.name: com.example.kafka.tiered.storage.s3.S3RemoteStorageManager
remote.log.storage.manager.class.path: /opt/kafka/plugins/tiered-storage-s3/*
remote.log.storage.manager.impl.prefix: rsm.config.
rsm.config.storage.bucket.name=my-bucket
# Custom RLMM configs
rlmm.config.remote.log.metadata.topic.replication.factor: 1
There are certain limitations for a tiered storage enabled cluster. At the time of this proposal writing, these are the known limitations:
- Tiered storage is not supported on clusters utilizing the multiple Log directories on a broker (JBOD feature).
- Currently, tiered storage is not available for compacted topics.
- Once Tiered Storage is enabled for a topic, it becomes a permanent configuration and cannot be disabled.
- Disabling Tiered Storage for the entire cluster requires manual deletion of all topics using Tiered Storage.
- All Kafka clients, regardless of their version, can continue to produce and consume records from topics utilizing Tiered Storage. However, clients with versions earlier than 3.0 are limited in performing administrative actions, such as enabling Tiered Storage on a topic
We choose not to validate the error conditions at the Strimzi operator layer because:
- At least some of them are expected to be removed in the future
- Validating them might be quite complex (for example, IIRC, the compacted topics can be set in the topic itself as well as in the broker defaults)
- Validating this in UTO would capture only a part of the topics and not prevent the users from misconfiguration. So they will still need to be aware of these rules and learn to respect them.
This proposal doesn't include built in plugin support in any Strimzi built Kafka image. Users need to specify the path via the config, and customize the docker image to include the library.
An example to build such custom image is via a custom Dockerfile:
FROM quay.io/strimzi/kafka:0.40.0-kafka-3.6.0
ARG REMOTE_PLUGIN_PATH
COPY ${REMOTE_PLUGIN_PATH} /opt/kafka/plugins/tiered-storage-s3/
An image with plugin library can be built with the following command:
docker build -t docker.example.com/tierstorage/kafka:0.40.0-kafka-3.6.0-plugin-1 . -build-arg REMOTE_PLUGIN_PATH=./s3/build/install/s3/*
Unit tests will be added to validate that all the RLM, RSM, and RLMM changes are propagated through. Because the RSM and RLMM are typically provided by user and there is no built in support for any third party cloud provider, there won’t be any integration tests covering the cloud provider interaction. It’s the user’s responsibility to ensure the RSM and RLMM works with the intended cloud provider.
For testing purposes, a system test with LocalTieredStorage implementation can be added to ensure end-to-end functionality.
Strimzi operator will be affected with the change.
This proposal adds a new API and configuration, so there are no backward compatibility issues with the existing APIs.
A simple way is to set everything via Helm chart config: #98
That option is not favored because it's not applicable to usage without Helm chart. It might also not be able to guarantee compatibility with future Strimzi versions.
A optional idea during review discussion is to consider adding a feature gate to protect the Tiered Storage feature:
- If it is disabled, the tieredStorage section from the Kafka CR will be essentially ignored By enabling it, users confirm they understand the limitations
- Once it moves out of early access in Kafka and the limitations are fixed, we can enable the feature gate by default.
This idea is not included in this proposal because having the feature config setup assume the users to be familiar with the feature and its limitation. Having another feature gate doesn't add much value.
Another idea is to open the customization for RLMM in the config as well. This is not included right now because in most cases the TopicBasedRemoteStorageMetadataManager is the default choice. In the future if such requirement comes, we can easily extend the API to allow custom RLMM implementation.