Search code examples
pythonapache-kafkaavrokafka-consumer-apiconfluent-platform

Kafka python AvroConsumer seek method


I am having issues consuming from a given offset using python, in Java I had implemented a KafkaConsumer and used

consumer.seek(new TopicPartition(topic, 0), 3)

to consume from my third offset in partition 0.

But when using Python I am using an AvroConsumer, which has a seek method but only takes the partition as a parameter and not an offset to consume from. it seems to consume then from the beginning of the topic. Is there a way with AvroConsumer to consume from a certain offset? Thanks


Solution

  • It appears that you provide the offset as a parameter when creating the topic partition.

    I don't have an example of your code to model this off of so I'll just write it like this.

    confluent_kafka.TopicPartition(topic,
                                   partition,
                                   offset)
    
    consumer.seek(tp)
    

    This is based on the integration test in the library for this code. I have highlighted the relevant test and linked below

    Integration Test For Confluent Kafka Python