I would like to share what I found about this topic, I hope this can help people who come across this as well.
We all should know how to increase a topics replication factor by now, if you dont you can go here; https://kafka.apache.org/documentation/#basic_ops_increase_replication_factor
Lets say you have a cluster which is deployed and running with offset.topic.replication.factor=2, and this is not a dynamically updateable setting. So you decided to update each broker and controllers static configuration and increase your offset.topic.replication.factor from 2 to 3 and did a rolling restart.
Does this automatically increase the consumer offsets topic replication factor? In my case NO! Even when I tried the replication increase with reassignments script, the new replicas always disappeared.
So how do we solve this issue?
Since __consumer_offsets is a compact topic, I believe if your cluster do not have any active consumers you cannot increase the replication factor of __consumer_offsets.
The reason behind this, if I am not mistaken, when the new replicas created and has no logs coming, since we dont have any active consumer groups, when the compaction interval hits the replicas are being deleted. So, I can get the replication factor of consumer offsets from 2 to 3 with reassigning, but I see them disappear after a short while (2 minutes default).
How I managed to keep them with the cluster? It was pretty simple when I understood what was happening. Just make sure you have a console consumer set up somewhere, after you start the reassigment of replicas start your console consumer. Since this new console consumer will create a consumer group and its own consumer offsets, compact log will add the offsets for this new consumer group to __consumer_offsets and it will be replicated as needed.
After this I saw some very weird logs from broker saying they are updating some configurations with empty values, but after that my __consumer_offsets topic now has 3 replicas.
I hope this will help people.
Regards,