Search code examples
jsonsplitapache-nifijolt

Nifi - splitting root json elements into different flowfiles

I have a data source that is sending json formatted messages containing various key value pairs. It is not consistent which keys are included in each message. However, i need to split each individual key value pair onto its own flow file and be able to extract the key name

The reasoning for this, is that i need to perform a table lookup using the key name, to generate a new data format, including the value

For example, if i receive a message such as this:

{
  "key1": 123,
  "Key2": 4234,
  "Key3": 789  
}

I want the output to be three seperate flowfiles

{
  "key1": 123
}

{
  "Key2": 4234
}

{
  "Key3": 789 
}

I also need to know based on those individual flow files how i could extract the name of the key

Solution

  • Right,

    you need a JoltTransformJSON processor for this one

    You Flow will look like this : enter image description here

    Where my jolt setup it : enter image description here

    Spec for jolt:

    [
  {
    "operation": "shift",
    "spec": {
      "*": {
        "$": "[#2].Key",
        "@": "[#2].Value"
      }
    }
  }
]

    **Next you split the output on $.*.

    And Evaluate Key and Value using EvaluateJson , then you use the attributes in Lookup how ever you want.

    This would be the output of the Jolt Trans :

    [
  {
    "Key": "key1",
    "Value": 123
  },
  {
    "Key": "Key2",
    "Value": 4234
  },
  {
    "Key": "Key3",
    "Value": 789
  }
]