Mirror Maker active-active setup with consumers at both source and target cluster

[sandeeptengale]

I am trying to perform Kafka cluster migration from one cluster to another cluster using Mirror Maker 2.0.
I am using Apache Kafka 2.8.1 which has good support for consumer-group replication and offset sync. I have encountered a scenario where I want to migration a topic along with its producers and consumers from source cluster to target cluster.

Example: I have topic A which I want to migrate, it has 3 partitions

source -> topic = A
destination -> topic = source.A
topic A is replicated as source.A in target cluster

I have a consumer-group topic-a-reader-group created at source cluster with 3 consumers. It is replicated at destination cluster with same name, I have created 1 consumers in this group at destination cluster.

Now when I producer messages at source cluster to topic A, it is consumed by 3 consumers are source cluster and this message will also get replicated at target cluster which is consumer by the consumer present in the consumer-group running at target cluster. Basically the message is read both by source consumer and target consumer. Duplicate read for consumer ultimately.

I want only one consumer to get this message, not to duplicate at source and destination. In my application, I can't just turn off the consumer at source and move it to target cluster(time critical application). So I want to keep both consumers at source and target running for sometime and turn off source consumer after some duration where both are running stable.

Is there any idempotence property available for consumer-group which let only one consumer-group to read the message with mirror maker scenario (either source or target) without being duplicated at source and target cluster?

[scholzj]

That is not really how it works. The consumer groups on each cluster are independent on each other. So if the clients run on both sides, you will have duplicates and you will need to deal with them in your application. The typical migration is that you simply stop it on one side and start it on the other.

[sandeeptengale]

Thank @scholzj for the response. Is there any other way to get mirror maker working in bi-directional and moving consumer from source to target cluster with zero downtime (like rolling restart or something)?
Am referring this blog post to work around mirror maker
https://strimzi.io/blog/2020/03/30/introducing-mirrormaker2/
https://dalelane.co.uk/blog/?p=4074

[scholzj]

I think you are misunderstanding what the blog post says. The bi-directional replication covered there is about data only.