Search code examples
jsonapache-nifijolt

Jolt not printing anything


I am writing jolt for transforming this data but not getting desired result

If practice_loc,prac_num and topId are same for two or more data then they will be combined together with separate S1 and S2 within subList. Else they would pass as it is with addition of subList only.

Data

[
  {
    "practice_loc": "120",
    "prac_num": "oswal",
    "topId": "t1",
    "S1": "A1",
    "S2": "B1"
  },
  {
    "practice_loc": "120",
    "prac_num": "oswal",
    "topId": "t1",
    "S1": "A2",
    "S2": ""
  },
  {
    "practice_loc": "334",
    "prac_num": "L3",
    "topId": "plumcherry",
    "S1": "A3",
    "S2": ""
  },
  {
    "practice_loc": "987",
    "prac_num": "L3",
    "topId": "artica",
    "S1": "A5",
    "S2": "B7"
  }
]

Expected Output:

[
  {
    "practice_loc": "120",
    "prac_num": "oswal",
    "topId": "t1"
    "subList": [
      {
        "S1": "A1",
        "S2": "B1"
      },
      {
        "S1": "A2",
        "S2": ""
      }
    ]
  },
  {
    "practice_loc": "334",
    "prac_num": "L3",
    "topId": "plumcherry"
    "subList": [
      {
        "SubID1": "A3",
        "SubID2": ""
      }
    ]
  },
  {
    "practice_loc": "987",
    "prac_num": "L3",
    "topId": "artica",
    "subList": [
      {
        "SubID1": "A5",
        "SubID2": "B7"
      }
    ]
  }
]

Here is what I tried but didnt get desired result Its not printing anything

[
  {
    "operation": "shift",
    "spec": {
      "*": {
        "@": "@(1,practice_loc).@(1,prac_num).@(1,topId)"
      }
    }
  },
  {
    "operation": "cardinality",
    "spec": {
      "*": {
        "*": "MANY"
      }
    }
  },
  {
    "operation": "shift",
    "spec": {
      "*": {
        "*": {
          "*": {
            "practice_loc": "[#4].&",
            "prac_num": "[#4].&",
            "topId": "[#4].&",
            "S*": "[#4].subList[&1].&"
          }
        }
      }
    }
  },
  {
    "operation": "cardinality",
    "spec": {
      "*": {
        "practice_loc": "ONE",
        "prac_num": "ONE",
        "topId": "ONE"
      }
    }
  }
]

Solution

  • Your current spec is pretty good. Would be suitable to rearrange it like that

    [
      { // group by those three attributes
        "operation": "shift",
        "spec": {
          "*": {
            "*": "@1,practice_loc.@1,prac_num.@1,topId.&",
            "S*": "@1,practice_loc.@1,prac_num.@1,topId.subList[&1].&"
          }
        }
      },
      { // get rid of wrappers
        "operation": "shift",
        "spec": {
          "*": {
            "*": {
              "*": {
                "@": ""
              }
            }
          }
        }
      },
      {
        "operation": "cardinality",
        "spec": {
          "*": {
            "*": "ONE", // pick only single one from repeating components 
            "subList": "MANY"
          }
        }
      },
      { // get rid of generated nulls within subList arrays 
        "operation": "modify-overwrite-beta",
        "spec": {
          "*": "=recursivelySquashNulls"
        }
      }
    ]
    

    Edit for illustration : Below, I have pasted the image what I get after toggling ADVANCED tab of Configure section for the JoltTransformJSON processor which has the version 1.21.0 as NiFi does. Btw, yours is a recent version as well.

    enter image description here