Search code examples
jsonapache-nifiavro

Writing AVRO fixed type from JSON in NIFI

I'm trying to convert a NiFi flow file containing JSON to an AVRO record.

The problem I have is that I don't know how to deal with a fixed type in AVRO, i.e. how to specifiy the proper JSON for converting to fixed?

Currently I'm using the ConvertJsonToAvro-processor. The AVRO output schema:

{
  "type" : "record",
  "name" : "Message",
  "namespace" : "com.example",
  "fields" : [ {
    "name" : "MAC",
    "type" : {
      "type" : "fixed",
      "name" : "MY_FIXED_TYPE",
      "size" : 6
    }
  }]
}

The input JSON-forms I tried are

{ "MAC": [ 0, 1, 2, 3, 4, 5] }

{ "MAC": "012345" }

{"MAC":"\u0000\u0001\u0002\u0003\u0004\u0005"}

{"MAC":{"MY_FIXED_TYPE": "\u0000\u0001\u0002\u0003\u0004\u0005"}}

Unfortunately none of them worked for me. I also tried the ConvertRecord-processor instead of the ConvertJsonToAvro-processor. Also without any luck.

Any ideas?

Solution

  • After some further investigation, it looks like the ConvertJsonToAvro processor can't be used to generate Avro FIXED or BYTES datum. This is likely a bug with NiFi and how the processor uses Avro.

    If I'm not mistaken:

    1. The NiFi ConvertJsonToAvro uses the KiteSDK to interpret JSON into Avro data. This JSON-to-Avro conversion is not the same as Avro JSON encoding from the specification.
    2. This processor reads the incoming string into a jackson JsonNode.
    3. FIXED and BYTES types need to correspond to a JsonNode where isBinary() is true.
    4. As far as I can tell, parsing a JSON string with Jackson never generates such a JSON node.

    I would raise a NiFi JIRA about this, or an issue on the KiteSDK.