Search code examples
jsontransformapache-nifijolt

JOLT Transform : Extract and combine fields from multiple JSON arrays to single array output, where one of the arrays can also be null


I had asked a question previously: JOLT Transform : Extract and combine fields from multiple JSON arrays to single array output

This is a bit different question, where the arrays can actually consist null values.

Example 1:

{
  "questionResponses": [
    {
      "responses": null
    }
  ],
  "answers": [
    {
      "questionId": "1",
      "answerId": "answer 1"
    },
    {
      "questionId": "1",
      "answerId": "answer 2"
    }
  ],
  "totalAttachments": 0
}

Undesired Output:

null

Expected Output:

[
  {
    "response": null,
    "answerId": "answer 1"
  },
  {
    "response": null,
    "answerId": "answer 2"
  }
]

Example 2:

{
  "questionResponses": [
    {
      "responses": [
        {
          "responseId": "1",
          "response": "response 1"
        },
        {
          "responseId": "2",
          "response": "response 2"
        },
        {
          "responseId": "3",
          "response": "response 3"
        }
      ]
    }
  ],
  "answers": null,
  "totalAttachments": 0
}

Undesired Output:

null

Expected Output:

[
  {
    "response": "response 1",
    "answerId": null
  },
  {
    "response": "response 2",
    "answerId": null
  },
  {
    "response": "response 3",
    "answerId": null
  }
]

JOLT Spec:

[
  {
    "operation": "shift",
    "spec": {
      "questionR*": {
        "*": {
          "responses": {
            "*": {
              "@(4,answers)": "@(1,response)"
            }
          }
        }
      }
    }
  },
  {
    // Determine two independent arrays both with size of 6
    "operation": "shift",
    "spec": {
      "*": {
        "*": {
          "$1": "response",
          "a*": "&"
        }
      }
    }
  },
  {
    // expand those arrays as objects which contain desired key-value pairs
    "operation": "shift",
    "spec": {
      "*": {
        "*": "&.&1"
      }
    }
  },
  {
    // get rid of object labels
    "operation": "shift",
    "spec": {
      "*": ""
    }
  }
]

The JOLT Spec works as expected when both of the arrays consists at least single value. But fails when either of them is null.


Solution

  • You can use shift transformations to call responses and questionResponses individually while referencing one nested in another mutually, in first of them

    (@(4,answers) nested within questionResponses

    and

    @(2,questionResponses) nested within answers)

    such as

    [
      {
        "operation": "shift",
        "spec": {
          "questionR*": {
            "*": {
              "responses": {
                "*": {
                  "response": "[&1].&",
                  "@(4,answers)": "[&1].answerId"
                }
              }
            }
          },
          "answers": {
            "*": {
              "@(2,questionResponses)": {
                "*": {
                  "*": "&4.&.[&2]"
                }
              },
              "ans*": {
                "@": "&3.&1[&2]"
              }
            }
          }
        }
      },
      {
        "operation": "shift",
        "spec": {
          "answers": {
            "*": {
              "*": {
                "@": "[&].&2"
              }
            }
          },
          "*": ""
        }
      }
    ]
    
    

    the demoes on the site http://jolt-demo.appspot.com/ :

    for Example 1:

    enter image description here

    for Example 2:

    enter image description here