Configure retention time

[ThePoipoi]

Hello all.
Just a quick question: I'm trying to configure my Kafka cluster to have a custom retention time.
I have found some information about it in the 0.5.0 documentation, but nothing since.

So I have tried to used default parameters from Kafka, as shown below:

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    version: 2.6.0
    replicas: 1
    [...]
    config:
      [...]
      log.retention.ms: 120000
      log.retention.hours: 0
      log.retention.bytes: 10737
      log.delete.retention.ms: 120000
      log.cleaner.delete.retention.ms: 120000
      log.retention.check.interval.ms: 10000
[...]

The goal is to remove each remaining topic message after a specified amount of time (in this case, I'm trying to keep messages only for 2 minutes).

Correct me if I'm wrong, but I have understand that the retention period is used only to marks data segments for deletion. And if I want thoses data delete, I have to use the "cleaner".
Am I right ? What do I do wrong (because nothing is never removed) ?

Thanks

[scholzj]

Kafka topics have two policies: delete and cleanup. With the cleanup policy, it is keeping the last message for given message key. With the delete policy, it will delete them based on time. But it will always only delete the whole log segments once all messages in them are over the retention time. You can have some control over how often that happens for example by configuring how often new segments are created (using log.roll.ms) so that the old segments are closed and can be deleted even when they are not full. But to do this so often might have performance impact.

The configuration you have above should work in general. But:

• Topics might still override these settings. They are basically just the defaults when not set in the topic.
• The log.retention.ms and log.retention.hours conflict with each other. Use just one of them.

But the settings you have there would not do what you want. If you want to have some strict time-to-live / message-expiration style behaviour, I do not think Kafka is the right solution.

[scholzj]

These are the partition directories ... you have to check the files inside them.

[ThePoipoi]

Sorry, I was not clear enough, but it's the same inside each folder

[kafka@my-cluster-kafka-0 kafka-log0]$ cd syncTopicSDE-0
[kafka@my-cluster-kafka-0 syncTopicSDE-0]$ ls
total 8
-rw-r--r--. 1 kafka root        0 Feb  1 10:41 00000000000000002120.log
-rw-r--r--. 1 kafka root       10 Feb  1 10:41 00000000000000002120.snapshot
-rw-r--r--. 1 kafka root 10485756 Feb  1 10:41 00000000000000002120.timeindex
-rw-r--r--. 1 kafka root       11 Feb  1 10:41 leader-epoch-checkpoint

And it's confirmed by this command:
kubectl -n kafka-a exec -ti my-cluster-kafka-0 -- ./bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-tls-bootstrap:9093 --topic syncTopicSDE --from-beginning

Because it still list each message since this morning 10:41

But another thing: when I push more messages to this specific topic (syncTopicSDE), a new file is created in partition directories, named as the ".log" file but tagged as ".index".
This specific file is alive for 2/3 min then disapears.

[scholzj]

I guess the main question is whether these change every 2 minutes as you have it in configs now. Is it always only this 2120 file? Or are new files being created and deleted?

One thing to keep in mind is that by default, the clients can set different timestamps. And if one of your clients sets a timestamp in the future, it might block the deletion. That can be controlled by log.message.timestamp.difference.max.ms and log.message.timestamp.type. Not sure it is your case or not of course.

I'm not sure I understand what does it mean new file is created in partition directories, named as the ".log" file but tagged as ".index". But if it is deleted after 3-2 minutes that almost sounds like what you wanted? The new segements to be created and deleted often?

[scholzj]

Glad it works now! 👍

[ThePoipoi]

OK !!!

So finally, it seems to work. I wasn't understanding how files were managed, and I misunderstood information...
But it works ! If I run the previous command with "from-beginning", finally they disappeared. Not exactly after 2 min like you said, but a little bit more, waiting for the next loop to work (and don't worry, it's not a problem for us). I also think I've been confused by the timestamp value.

In fact, I have observed how the folder is managed right after sending new messages to Kafka:

• a new file 00000000000000004158.index is created
• after a few time, 3 more files are created (00000000000000004158.index.deleted 00000000000000004158.log.deleted and 00000000000000004158.timeindex.deleted). I assume that they have been tag as "can be deleted"
• after another time, all of those files are removed, and nothing is remaining in the topic (if I run an new consumer, it just waiting); just "LOG-END-OFFSET" are still remaining as previously with a bigger index

So so so sorry for losing your time, it was all my bad. Since I made the modification with log.roll.ms, everything is fine. I just mixed some information and made mistakes...