Jolt transform to remove duplicate values in the array

Question

I need to remove duplicates from the below JSon, if there are duplicates in the closed Array, only one value should be retained.

In the below example, the Closed Array contains PP21 two times, i want only once. like [ "PP21"]

Input

[
  {
    "name": "SAndOP-ProductionPlanningTable",
    "dataIO": {
      "data": [
        {
          "plant": "1000",
          "material": "000002074099700010",
          "qty": 15
        },
        {
          "plant": "1000",
          "material": "000002074099700010",
          "qty": 15
        },
        {
          "plant": "1000",
          "material": "000002074099700010",
          "qty": 15
        }
      ],
      "Closed": [ "PP21", "PP21" ]
    }
  },
  {
    "name": "SAndOP-ProductionPlanningTable2",
    "dataIO": {
      "data": [
        {
          "plant": "1000",
          "material": "000002074099700010",
          "qty": 15
        },
        {
          "plant": "1000",
          "material": "000002074099700010",
          "qty": 15
        }
      ],
      "Closed": [ "PP22", "PP23" ]
    }
  }
]

Output Expected

[
  {
    "name": "SAndOP-ProductionPlanningTable",
    "dataIO": {
      "data": [
        {
          "plant": "1000",
          "material": "000002074099700010",
          "qty": 15
        },
        {
          "plant": "1000",
          "material": "000002074099700010",
          "qty": 15
        },
        {
          "plant": "1000",
          "material": "000002074099700010",
          "qty": 15
        }
      ],
      "Closed": [ "PP21" ]
    }
  },
  {
    "name": "SAndOP-ProductionPlanningTable2",
    "dataIO": {
      "data": [
        {
          "plant": "1000",
          "material": "000002074099700010",
          "qty": 15
        },
        {
          "plant": "1000",
          "material": "000002074099700010",
          "qty": 15
        }
      ],
      "Closed": [ "PP22", "PP23" ]
    }
  }
]

I have looked at examples and didn't get any closer as to what is mentioned above. I need to transform a JSON structure by using a JOLT spec. I use https://jolt-demo.appspot.com to test the following below. Can someone please suggest how I can get this to work. Thanks in advance

Answer 1

You can make the keys unique with null values such as

"Closed" : {
        "PP21" : null
      }

by using

"Closed": {
            "*": {
              "*": "......&"
            }
          }

match logic

and then tidy them up by

"$": "....&2[]"

matching in order to pick the respective keys only as the "Closed" array's components.

So, use the following transformation :

[
  {
    "operation": "shift",
    "spec": {
      "*": {
        "*": "&1.&", //the elements other than "dataIO"
        "dataIO": {
          "*": "&2.&1.&", //the elements other than "Closed"
          "Closed": {
            "*": {
              "*": "&4.&3.&2.&"
            }
          }
        }
      }
    }
  },
  {
    "operation": "shift",
    "spec": {
      "*": {
        "*": "[&1].&",
        "dataIO": {
          "*": "[&2].&1.&",
          "Closed": {
            "*": {
              "$": "[&4].&3.&2[]"
            }
          }
        }
      }
    }
  }
]

the demo on the site https://jolt-demo.appspot.com/ is :