Publishing Avro messages using Kafka REST Proxy throws "Conversion of JSON to Avro failed"

Question

I am trying to publish a message which has a union for one field as

{
  "name": "somefield",
  "type": [
    "null",
    {
      "type": "array",
      "items": {
        "type": "record",

Publishing the message using the Kafka REST Proxy keeps throwing me the following error when somefield has an array populated.

{
  "error_code": 42203,
  "message": "Conversion of JSON to Avro failed: Failed to convert JSON to Avro: Expected start-union. Got START_ARRAY"
}

Same schema with somefield: null is working fine.

The Java classes are built in the Spring Boot project using the gradle plugin from the Avro schemas. When I use the generated Java classes and publish a message, with the array populated using the Spring KafkaTemplate, the message is getting published correctly with the correct schema. (The schema is taken from the generated Avro Specific Record) I copy the same json value and schema and publish via REST proxy, it fails with the above error.

I have these content types in the API call

accept:application/vnd.kafka.v2+json, application/vnd.kafka+json, application/json
content-type:application/vnd.kafka.avro.v2+json

What am I missing here? Any pointers to troubleshoot the issue is appreciated.

Answer 1

The messages I tested for were,

{
   "somefield" : null
}

and

{
   "somefield" : [
      {"field1": "hello"}
    ]
}

However, it should be instead passed as,

{
   "somefield" : {
      "array": [
      {"field1": "hello"}
    ]}
}