How to encode/decode Kafka messages using Avro binary encoder?

Question

I'm trying to use Avro for messages being read from/written to Kafka. Does anyone have an example of using the Avro binary encoder to encode/decode data that will be put on a message queue?

I need the Avro part more than the Kafka part. Or, perhaps I should look at a different solution? Basically, I'm trying to find a more efficient solution to JSON with regards to space. Avro was just mentioned since it can be more compact than JSON.

Answer 1

I finally remembered to ask the Kafka mailing list and got the following as an answer, which worked perfectly.

Yes, you can send messages as byte arrays. If you look at the constructor of the Message class, you will see -

def this(bytes: Array[Byte])

Now, looking at the Producer send() API -

def send(producerData: ProducerData[K,V]*)

You can set V to be of type Message and K to what you want your key to be. If you don't care about partitioning using a key, then set that to Message type as well.

Thanks, Neha