Search code examples
apache-nifi

NiFi: JoltTransformJSON specification

I have following JSON:

{
  "results": [
    {
      "customerClient": {
        "resourceName": "customers/7876562723/customerClients/8506630423",
        "clientCustomer": "customers/8506630423",
        "hidden": false,
        "level": "1",
        "manager": false,
        "descriptiveName": "BMW",
        "id": "85061423"
      }
    },
    {
      "customerClient": {
        "resourceName": "customers/7876562723/customerClients/6736523142",
        "clientCustomer": "customers/6736523142",
        "hidden": false,
        "level": "1",
        "manager": true,
        "descriptiveName": "Mercedes",
        "id": "67363142"
      }
    }
}
  ],
  "fieldMask": "customerClient.clientCustomer,customerClient.hidden,customerClient.level,customerClient.descriptiveName,customerClient.id,customerClient.manager"
}

What i want:

  1. remove fieldMask parameter, so i don't need results as array.
  2. remove results and customerCient. So my JSON should look like this:
{
   "resourceName": "customers/7876562723/customerClients/8506630423",
   "clientCustomer": "customers/8506630423",
   "hidden": false,
   "level": "1",
   "manager": false,
   "descriptiveName": "BMW",
   "id": "85061423"
},
{
   "resourceName": "customers/7876562723/customerClients/6736523142",
   "clientCustomer": "customers/6736523142",
   "hidden": false,
   "level": "1",
   "manager": true,
   "descriptiveName": "Mercedes",
   "id": "67363142"
}

To remove results i'm using SplitJson with split value: $.results. And now i'm stuck at modifying it with JOLT specification. How can I do described above actions with only JoltTransformJSON?

Solution

  • Ok, so I believe JOLT can only output one object, so transforming each element of the array to unique objects wouldn't be possible in one JOLT. However, you can get most of the way there without putting FlowFile data in to Attributes.

    We can:

    • delete fieldMask
    • Remove the results array level

    Use this to test out JOLTs: https://jolt-demo.appspot.com/#inception

    Assuming your JSON is actually valid, so:

    {
  "results": [
    {
      "customerClient": {
        "resourceName": "customers/7876562723/customerClients/8506630423",
        "clientCustomer": "customers/8506630423",
        "hidden": false,
        "level": "1",
        "manager": false,
        "descriptiveName": "BMW",
        "id": "85061423"
      }
    },
    {
      "customerClient": {
        "resourceName": "customers/7876562723/customerClients/6736523142",
        "clientCustomer": "customers/6736523142",
        "hidden": false,
        "level": "1",
        "manager": true,
        "descriptiveName": "Mercedes",
        "id": "67363142"
      }
    }
  ],
  "fieldMask": "customerClient.clientCustomer,customerClient.hidden,customerClient.level,customerClient.descriptiveName,customerClient.id,customerClient.manager"
}

    You could use this JOLT spec:

    [
  {
    "operation": "remove",
    "spec": {
      "fieldMask": ""
    }
  },
  {
    "operation": "shift",
    "spec": {
      "results": {
        "*": {
          "customerClient": "customerClient-&1"
        }
      }
    }
  }
]

    Which gives you following result:

    {
  "customerClient-0" : {
    "resourceName" : "customers/7876562723/customerClients/8506630423",
    "clientCustomer" : "customers/8506630423",
    "hidden" : false,
    "level" : "1",
    "manager" : false,
    "descriptiveName" : "BMW",
    "id" : "85061423"
  },
  "customerClient-1" : {
    "resourceName" : "customers/7876562723/customerClients/6736523142",
    "clientCustomer" : "customers/6736523142",
    "hidden" : false,
    "level" : "1",
    "manager" : true,
    "descriptiveName" : "Mercedes",
    "id" : "67363142"
  }
}

    So you can transform your JSON to a flat structure using just the JOLT, then you could use SplitJSON to break each object up (if needed).

    You should consider using Records instead of SplitJSON, this would probably be more efficient.

    Read about records:

    Another interesting alternative could be ScriptedTransformRecord