Jolt flattening of array introduces unwanted null array item

Question

we have a requirement to flatten a JSON object. We have input JSON data that looks like this:

{
  "changes": [
    {
      "l1_Entity_Id": 7004,
      "l1_Entity_Nm": "Academic Administration2",
      "visibility": false,
      "L2_Entities": [
        {
          "L2_Entity_Id": 8003,
          "L2_Entity_Nm": "Desktop Software2",
          "visibility": 1,
          "primary_triage": "EAAS",
          "markham_triage": "MK_Team1",
          "Faculty_Support": [
            "Faculty1",
            "Faculty2",
            "Faculty4"
          ],
          "hasReference": true
        }
      ]
    },
    {
      "l1_Entity_Id": 7002,
      "l1_Entity_Nm": "Telephony5",
      "visibility": false,
      "L2_Entities": [
        {
          "L2_Entity_Id": 8005,
          "L2_Entity_Nm": "Automatic Call Distribution3",
          "visibility": 1,
          "primary_triage": "Telecom",
          "markham_triage": null,
          "Faculty_Support": [
            "Faculty3",
            "Faculty4"
          ],
          "hasReference": true
        },
        {
          "L2_Entity_Id": 8004,
          "L2_Entity_Nm": "Phone",
          "visibility": 1,
          "primary_triage": null,
          "markham_triage": null,
          "Faculty_Support": [
            "Faculty1"
          ],
          "hasReference": true
        }
      ]
    }
  ],
  "methodName": "generateEntitiesChgReport"
}

Once shifted, we'd like output data like this:

[
    {
        "l1_Entity_Nm": "Academic Administration2",
        "L1_visibility": false,
        "L2_Entity_Id": 8003,
        "L2_Entity_Nm": "Desktop Software2",
        "aggregateNm": "Academic Administration2>Desktop Software2",
        "L2_visibility": 1,
        "primary_triage": "EAAS",
        "isMarkham": true,
        "markham_triage": "MK_Team1",
        "Faculty_Support": [
            "Faculty1",
            "Faculty2",
            "Faculty4"
        ]
    },
    {
        "l1_Entity_Nm": "Telephony5",
        "L1_visibility": false,
        "L2_Entity_Id": 8005,
        "L2_Entity_Nm": "Automatic Call Distribution3",
        "aggregateNm": "Telephony5>Automatic Call Distribution3",
        "L2_visibility": 1,
        "primary_triage": "Telecom",
        "isMarkham": false,
        "markham_triage": null,
        "Faculty_Support": [
            "Faculty3",
            "Faculty4"
        ]
    },
    {
        "l1_Entity_Nm": "Telephony5",
        "L2_Entity_Id": 8004,
        "L2_Entity_Nm": "Phone",
        "aggregateNm": "Telephony5>Phone",
        "L2_visibility": 1,
        "primary_triage": null,
        "isMarkham": false,
        "markham_triage": null,
        "Faculty_Support": [
            "Faculty1"
        ]
    }
]

We almost got the desired output with the Jolt spec below:

[
  {
    "operation": "shift",
    "spec": {
      "changes": {
        "*": {
          "L2_Entities": {
            "*": {
              "@": "&[&3]",
              "@(2,l1_Entity_Id)": "&[&3].l1_Entity_Id",
              "@(2,l1_Entity_Nm)": "&[&3].l1_Entity_Nm",
              "@(2,l1_visibility)": "&[&3].l1_visibility"
            }
          }
        }
      }
    }
},
  {
    "operation": "shift",
    "spec": {
      "*": {
        "*": "[]"
      }
    }
}
]

However, it introduces an unwanted null after second json object as shown below:

[ {
  "L2_Entity_Id" : 8003,
  "L2_Entity_Nm" : "Desktop Software2",
  "visibility" : 1,
  "primary_triage" : "EAAS",
  "markham_triage" : "MK_Team1",
  "Faculty_Support" : [ "Faculty1", "Faculty2", "Faculty4" ],
  "hasReference" : true,
  "l1_Entity_Id" : 7004,
  "l1_Entity_Nm" : "Academic Administration2"
}, {
  "L2_Entity_Id" : 8005,
  "L2_Entity_Nm" : "Automatic Call Distribution3",
  "visibility" : 1,
  "primary_triage" : "Telecom",
  "markham_triage" : null,
  "Faculty_Support" : [ "Faculty3", "Faculty4" ],
  "hasReference" : true,
  "l1_Entity_Id" : 7002,
  "l1_Entity_Nm" : "Telephony5"
}, null, {
  "L2_Entity_Id" : 8004,
  "L2_Entity_Nm" : "Phone",
  "visibility" : 1,
  "primary_triage" : null,
  "markham_triage" : null,
  "Faculty_Support" : [ "Faculty1" ],
  "hasReference" : true,
  "l1_Entity_Id" : 7002,
  "l1_Entity_Nm" : "Telephony5"
} ]

I hope someone can help us to resolve the issue. Thank you all for any help you might give us.

Answer 1

You can rather use * symbol than @ symbol along with non-square-bracketed ampersands such as

[
  {
    "operation": "shift",
    "spec": {
      "changes": {
        "*": {
          "L2_Entities": {
            "*": {
              "*": "&3.&1.&",
              "@2,l1_Entity_Id": "&3.&1.l1_Entity_Id",
              "@2,l1_Entity_Nm": "&3.&1.l1_Entity_Nm",
              "@2,l1_visibility": "&3.&1.l1_visibility"
            }
          }
        }
      }
    }
  },
  { // get rid of the object keys
    "operation": "shift",
    "spec": {
      "*": {
        "*": ""
      }
    }
  }
]

or you can make the transformation more dynamic in which no need to state each attributes individually such as

[
  {
    "operation": "shift",
    "spec": {
      "*": {
        "*": { // loop through all the "changes"
          "l1*": "&2.&1.Others.&", // accumulate the elements other than "L2_Entities" and "visibility" within an individual object caled "Others"
          "L2_Entities": "&2.&1.&"
        }
      }
    }
  },
  {
    "operation": "shift",
    "spec": {
      "*": {
        "*": {
          "L2_Entities": {
            "*": {
              "@2,Others": { "*": "&4.&2.&" }, // go two levels up the tree to grab the whole values from the "Others" object
              "*": "&3.&1.&"
            }
          }
        }
      }
    }
  },
  { // get rid of the object keys
    "operation": "shift",
    "spec": {
      "*": {
        "*": ""
      }
    }
  }
]

the demo 2 :