Search code examples
elasticsearchapache-kafkaavroconfluent-platform

kafka avro elasticsearch consumer allow unknown keys

Is there a way to allow Kafka-Avro to accept all keys from the producer, even when they aren't defined in the producers schema? Currently I'm running the producer with 

./bin/kafka-avro-console-producer --broker-list localhost:9092 --topic elasticsearch-sink --property value.schema='{"type":"record","name":"myRecord", "fields":[{"name":"esKey", type":"string"}]}'

And passing in
{"esKey":"keyExample", "undefinedKey": "empty"}

The Consumer will only show
{"esKey":"keyExample"}

And so the undefined key won't be sent to ElasticSearch either. I'd rather not define the schema due to the input being very large and not predictable.

Edit: Working with Confluent 4.1.1

Solution

  • Your schema doesn't allow the other field

    "fields":[{"name":"esKey", type":"string"}]

    You need to add undefinedKey into this for that field to appear. Basically, the Avro deserializer doesn't have a reason to think that the other field should exist.

    I'd rather not define the schema due to the input being very large and not predictable.

    Not clear what this means, but if you have unpredictable data, then perhaps Avro, or any strictly defined schema protocol, isn't the best choice?