JOLT Transform : Extract and combine fields from multiple JSON arrays to single array output

Question

My JSON object:

{
  "questionResponses": [
    {
      "responses": [
        {
          "responseId": "1",
          "response": "response 1"
        },
        {
          "responseId": "2",
          "response": "response 2"
        },
        {
          "responseId": "3",
          "response": "response 3"
        }
      ]
    }
  ],
  "answers": [
    {
      "questionId": "1",
      "answerId": "answer 1"
    },
    {
      "questionId": "1",
      "answerId": "answer 2"
    }
  ],
  "totalAttachments": 0
}

I need to have the output in following format:

[
  {
    "response": "response 1",
    "answerId": "answer 1"
  },
  {
    "response": "response 1",
    "answerId": "answer 2"
  },
  {
    "response": "response 2",
    "answerId": "answer 1"
  },
  {
    "response": "response 2",
    "answerId": "answer 2"
  },
  {
    "response": "response 3",
    "answerId": "answer 1"
  },
  {
    "response": "response 3",
    "answerId": "answer 2"
  }
]

I only have a solution to fetch fields from both of the arrays, but unsure how to join them without having any common key and represent them in required way

Current JOLT Spec:

[
  {
    "operation": "shift",
    "spec": {
      "questionResponses": {
        "*": {
          "responses": {
            "*": {
              "response": "responses.[&1].response"
            }
          }
        }
      },
      "answers": {
        "*": {
          "answerId": "answers.[&1].answerId"
        }
      }
    }
  }
]

Current undesired output:

{
  "responses": [
    {
      "response": "response 1"
    },
    {
      "response": "response 2"
    },
    {
      "response": "response 3"
    }
  ],
  "answers": [
    {
      "answerId": "answer 1"
    },
    {
      "answerId": "answer 2"
    }
  ]
}

Now I have to combine those responses with each answerId to get the desired output.

Also I'm using Apache Nifi for the solution, is there any other method/processor we can use if JOLT transform doesn't work as expected?

Answer 1

You can use recursive shift transformations starting by obtaining key-value pairs under answers array while strolling through the sub-attributes of responses array such as

[
  {
    "operation": "shift",
    "spec": {
      "questionR*": {
        "*": {
          "responses": {
            "*": {
              "@(4,answers)": "@(1,response)"
            }
          }
        }
      }
    }
  },
  { 
   // Determine two independent arrays both with size of 6
    "operation": "shift",
    "spec": {
      "*": {
        "*": {
          "$1": "response",
          "a*": "&"
        }
      }
    }
  },
  {
   // expand those arrays as objects which contain desired key-value pairs
    "operation": "shift",
    "spec": {
      "*": {
        "*": "&.&1"
      }
    }
  },
  {
   // get rid of object labels
    "operation": "shift",
    "spec": {
      "*": ""
    }
  }
]