How to fix a corrupted log segment?

[iosifnicolae2]

Hi.

We've deployed Strimzi in a K8S cluster that's using Longhorn as storage engine.

2022-12-07 15:38:40,219 ERROR [ReplicaManager broker=2] Error processing fetch with max size 1048576 from replica [1] on partition raw-teltonika-packets-6: PartitionData(fetchOffset=217064489, logStartOffset=0, maxBytes=1048576, currentLeaderEpoch=Optional[29], lastFetchedEpoch=Optional[25]) (kafka.server.ReplicaManager) [data-plane-kafka-request-handler-2]
--
Wed, Dec 7 2022 5:38:40 pm | org.apache.kafka.common.errors.CorruptRecordException: Found record size 0 smaller than minimum record overhead (14) in file /var/lib/kafka/data-0/kafka-log2/raw-teltonika-packets-6/00000000000216481511.log.

• an article with a description of a similar problem: https://stacksoft.io/blog/kafka-troubles/

bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files /var/lib/kafka/data-0/kafka-log2/raw-teltonika-packets-6/00000000000216481511.log

• first invalid record (position: 352321223, size: 329):

14FFFFB8   39 5A 22 2C  22 66 72 6F  6D 22 3A 22  73 65 72 76  65 72 22 2C  22 74 6F 22  3A 22 63 6C  69 65 6E 74  22 2C 22 64  61 74 61 5F  68 65 78 22  9Z","from":"server","to":"client","data_hex"
14FFFFE4   3A 22 30 30  30 30 30 30  30 38 22 2C  22 72 65 6D  6F 74 65 41  64 64 72 65  73 73 22 3A  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  :"00000008","remoteAddress":................
15000010   00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  ............................................
1500003C   00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  ............................................
15000068   00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  ............................................
15000094   00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  ............................................
150000C0   00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  ............................................

• next possible valid record (position: 1581676C)

15816714   00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  ............................................
15816740   00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  ............................................
1581676C   00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00  0C F0 52 43  00 00 01 3D  00 00 00 19  02 38 4E 94  22 00 00 00  00 00 00 00  00 01 84 EB  ..................RC...=.....8N."...........
15816798   08 EB 5F 00  00 01 84 EB  08 EB 5F FF  FF FF FF FF  FF FF FF 00  00 00 00 00  00 00 00 00  01 94 04 00  00 00 1E 12  12 32 36 12  35 12 39 12  .._......._........................35

Question

1. Is there a way to copy a partition from a topic to another topic using Strimzi config?
2. Is there a way in Kafka to reset the latest offset of a topic in Kafka?
3. Is there a way to fix the corrupted log segment?
4. What happens if I delete the corrupted log segment?
5. What happens if I delete all the log segments to the end starting with the first corrupted log segment?

[scholzj]

Don't you have some replicas to recover from? If it is corrupted on the storage level, I would expect it to be corrupted only in one replica. So you can just delete the partition on the broker where it corrupted and let it recover from the other replicas.

---

Re 1) No. Strimzi just deploys Apache Kafka. It is not a fork or some completely different application implementing the Kafka API. So basically all your questions are about Apache Kafka and have only little to do with Strimzi. So in general, whatever you can do with Kafka can be done with Strimzi. But there is no special magic with which would Strimzi allow you to mix and match parts of topics together. I suspect there is nothing like that in Kafka either. But I might be wrong about that.

Re 2) Well, the consumed / unconsumed offset is tracked per consumer group. You can use the Kafka Admin API or the kafka-consumer-groups.sh CLI tool to change the last committed offset for given consumer group.

Re 5) I never tried it from that side - from the newest segment. I did it always only from the oldest one in the past. I guess it might work from newest one as well - but again, not 100% sure. You need to delete all the segment files (i.e. also the indexes etc.)

[scholzj]

PS: There is also the kafka-delete-records.sh tool. I never used it. But I guess that might help to get rid of the corrupted messages as well?

[iosifnicolae2]

The problem is that the log was corrupted before being sync'ed to a replica and now we don't have any other replicas to recover from.

What would happen if we delete the corrupted logs and also all the following logs? (it's ok if we lose the last portion of the data).

• another solution would be to promote a replica as leader, so the data from the master replica will be lose (I assume)

Thank you very much for you response @scholzj !

[scholzj]

The problem is that the log was corrupted before being sync'ed to a replica and now we don't have any other replicas to recover from.

How exactly would that happen? As in corrupted memory?

another solution would be to promote a replica as leader, so the data from the master replica will be lose (I assume)

Not really - you would still need to delete the broken data from thedisk and let them resync. Otherwise the problem might still pop-up later.

[iosifnicolae2]

I don't know what caused the problem..

PS. I've managed to solve the problem by following the next steps:

1. backup valid records from the broken segment log
2. promote another replica as leader
3. delete the broken segment from disk
4. restore the valid records (even though, these records might not be in the right order..)