Confluent Kafka Python library configure producer for bulk msg

Question

I need to setup Kafka producer to send 500 msgs at one batch not msg by msg but bulk import. I checked https://github.com/dpkp/kafka-python/issues/479 and tried producer.send_messages(topic, *message) but it fails with error:

>       producer.send_messages('event_connector_mt', *load_entries)
E       AttributeError: 'cimpl.Producer' object has no attribute 'send_messages'

I also tried to pass it like producer.produce(topic, *message) fails with:

       producer.produce('event_connector_mt', *load_entries)
E       TypeError: function takes at most 8 arguments (501 given)

So i dig more and found out that i have to set in producer configuration the type to be async and batch.size to be bigger than default, but when i try config like:

from confluent_kafka import Consumer, Producer       

producer = Producer(**{'bootstrap.servers': KAFKA_BROKERS,
                       'queue.buffering.max.messages': 1000000,
                       'batch.num.messages': 500,
                       'batch.size': 19999,
                       'producer.type': 'async'
                       })

fails with:

E       KafkaException: KafkaError{code=_INVALID_ARG,val=-186,str="No such configuration property: "producer.type""}

same error for batch.size Can you pint me where and how i can setup async and batch size or any other way to pass bulk msgs to Kafka 0.9.3.1

Answer 1

All producers are by default asynchronous. Producer.type and batch.size are not supported configuration by underlying librdkafka library.

Therefore, please use the available configurations batch.num.messages or message.max.bytes.