Search code examples
jsonodataapache-nifi

Error trying to parse odata4 from API REST using NIFI

I'm using a Microsoft REST API to query a Azure application, oauth and request goes without problem.

The response from InvokeHTTP has this format

{"@odata.context":"https://****.dynamics.com/api/data/v9.1/$metadata#endpoint","value":[ here comes the actual JSON result in format {
  "@odata_etag" : "W/\"555598\"", "field":"value...},...]
,"@odata.nextLink":"https://****.dynamics.com/api/data/v9.1/endpoint?$skiptoken.....}

I need to extract the nextLink for pagination and Value to continue the flow and store the result.

When I try to parse with inferAvroSchema so I can start working with, it throws this error:

Illegal initial character: @odata.etag

My Idea was to inferAvroSchema, then EvaluateJsonPath to extract the odata tags and then extract the values.

I tried using EvaluateJsonPath on the result asking to create an attribute for [email protected] but it doesn't find the item either, I'm sure is something about the @.

I can also replace all the @ of the incoming flow for another char, but don't know if that makes sense.

I'm feeling that I'm not using a correct approach, but NIFI + odata doesn't give me results on google or here. I'm open to any suggestions!

thank you!

Solution

  • Schema fields cannot contain @. You could replace the @, however you must be sure not to replace it in actual content like email addresses. Another solution is to transform the API response using JoltTransformJSON processor, such that your flow can work with it:

    enter image description here

    GenerateFlowFile:

    enter image description here

    For the JoltTransformJSON processor provide following Jolt specification:

    [
  {
    "operation": "shift",
    "spec": {
      "\\@odata.nextLink": "next"
    }
  }
]

    Leave the default values for the other properties. You can play around with Jolt here: http://jolt-demo.appspot.com/

    EvaluateJsonPath:

    enter image description here

    Result:

    enter image description here

    Notice that the url is now part of the flowfile attributes.