I am writing jolt for transforming this data but not getting desired result
If practice_loc
,prac_num
and topId
are same for two or more data then they will be combined together with separate S1 and S2 within subList
. Else they would pass as it is with addition of subList only.
Data
[
{
"practice_loc": "120",
"prac_num": "oswal",
"topId": "t1",
"S1": "A1",
"S2": "B1"
},
{
"practice_loc": "120",
"prac_num": "oswal",
"topId": "t1",
"S1": "A2",
"S2": ""
},
{
"practice_loc": "334",
"prac_num": "L3",
"topId": "plumcherry",
"S1": "A3",
"S2": ""
},
{
"practice_loc": "987",
"prac_num": "L3",
"topId": "artica",
"S1": "A5",
"S2": "B7"
}
]
Expected Output:
[
{
"practice_loc": "120",
"prac_num": "oswal",
"topId": "t1"
"subList": [
{
"S1": "A1",
"S2": "B1"
},
{
"S1": "A2",
"S2": ""
}
]
},
{
"practice_loc": "334",
"prac_num": "L3",
"topId": "plumcherry"
"subList": [
{
"SubID1": "A3",
"SubID2": ""
}
]
},
{
"practice_loc": "987",
"prac_num": "L3",
"topId": "artica",
"subList": [
{
"SubID1": "A5",
"SubID2": "B7"
}
]
}
]
Here is what I tried but didnt get desired result Its not printing anything
[
{
"operation": "shift",
"spec": {
"*": {
"@": "@(1,practice_loc).@(1,prac_num).@(1,topId)"
}
}
},
{
"operation": "cardinality",
"spec": {
"*": {
"*": "MANY"
}
}
},
{
"operation": "shift",
"spec": {
"*": {
"*": {
"*": {
"practice_loc": "[#4].&",
"prac_num": "[#4].&",
"topId": "[#4].&",
"S*": "[#4].subList[&1].&"
}
}
}
}
},
{
"operation": "cardinality",
"spec": {
"*": {
"practice_loc": "ONE",
"prac_num": "ONE",
"topId": "ONE"
}
}
}
]
Your current spec is pretty good. Would be suitable to rearrange it like that
[
{ // group by those three attributes
"operation": "shift",
"spec": {
"*": {
"*": "@1,practice_loc.@1,prac_num.@1,topId.&",
"S*": "@1,practice_loc.@1,prac_num.@1,topId.subList[&1].&"
}
}
},
{ // get rid of wrappers
"operation": "shift",
"spec": {
"*": {
"*": {
"*": {
"@": ""
}
}
}
}
},
{
"operation": "cardinality",
"spec": {
"*": {
"*": "ONE", // pick only single one from repeating components
"subList": "MANY"
}
}
},
{ // get rid of generated nulls within subList arrays
"operation": "modify-overwrite-beta",
"spec": {
"*": "=recursivelySquashNulls"
}
}
]
Edit for illustration : Below, I have pasted the image what I get after toggling ADVANCED
tab of Configure
section for the JoltTransformJSON
processor which has the version 1.21.0 as NiFi does. Btw, yours is a recent version as well.