Search code examples
pythonapache-kafkakafka-producer-apiconfluent-platform

Confluent Kafka Python producer not producing with ACKS= all configuration


I have a some python code that will produce into a kafka topic, this works fine with the default setting acks=1 but when I change to acks=all or acks=2 the message does not end up in the topic. the min.insync.replicas config on the topic is set to 2. There is no error message returned after running the code which is confusing? There is 3 brokers in the cluster.

Here is the code

from confluent_kafka import Producer
from kafka.errors import KafkaError

def get_producer_config():
    return Producer(get_config())


def get_config():
    conf = {
        'bootstrap.servers': 'localhost:9092',
        'acks': '2',
    }

    return conf

try:
    producer = get_producer_config()
    producer.produce('test', 'test message from local app')
    producer.flush()
except KafkaError as error:
    get_logger().error(str(error))

This stems from debugging a kafka producer lambda where we were getting the error message KafkaError{code=NOT_ENOUGH_REPLICAS,val=19,str="Broker: Not enough in-sync replicas"} I was trying top replicate locally but a not getting the error but have noticed this while trying.

Thanks


Solution

    1. Acks: 2 - is not allowed.

    The number of acknowledgments the producer requires the leader to have received before considering a request complete. Allowed values: 0 for no acknowledgments, 1 for only the leader and -1 (or all) for the full ISR.

    1. You have to check , topic level configuration. Two parameters are relevant.
      min.insync.replicas(ISR) <= replication-factor