Kafka Mirror Maker - Lag is aggregated and then each 60 seconds it is zeroed and then all over again

Question

I am using Kafka MirrorMaker based on Kafka http://apache.cbox.biz/kafka/2.4.1/kafka_2.13-2.4.1.tgz image. My issue is that no matter what I tried, MirrorMaker is aggregating lag on 10 partitions of a topic for 60 seconds, then lag is zeroed in a second as messages are obviously not that much and then lag is growing again for 60 seconds. I would like messages and lag to be zeroed say each 10 seconds, but couldn't achieve it, although I played a bit mostly with mirrormaker producer config file.

consumer.props:

enable.auto.commit=false
client.id=mirror_maker_consumer
exclude.internal.topics=true
group.id=MirrorMaker
bootstrap.servers=${SOURCE_BOOTSTRAP_SERVERS_LIST}
partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor

producer.props:

acks=all
bootstrap.servers=${TARGET_BOOTSTRAP_SERVERS}
client.id=mirror_maker_consumer_telemetry
max.in.flight.requests.per.connection=1
retries=5
batch.size = 10
buffer.memory = 3000
linger.ms = 5
max.request.size = 1000
max.block.ms = 3000
receive.buffer.bytes = -1
send.buffer.bytes = -1
transaction.timeout.ms = 22000

On producer level I played with:

1. batch.size
2. linger.ms
3. max.request.size
4. tried adding bottom 4/5 properties to achieve less time between batches, no luck.

Some advice would be greatly appreciated.

I am not sure whether there is effective lag and messages remain on consumer side for those 60 seconds ot just lag is updated each 60 seconds with messages being replicate din the meantime...

Regards, OvivO

Answer 1

I have found the correct configuration and why consumer was committing offsets each 60 seconds. enable.auto.commit=false - this was my config and then consumer commits each 60 seconds.

If I deploy mirrormaker consumer with: enable.auto.commit=true auto.commit.interval.ms = 22000 it commits and clears lag each 22 seconds (22000 ms). However, I am not sure auto commit is safe so that no messages are lost.