Search code examples
jsonapache-nifijolt

Jolt transform - how to rename field with field value?

I'm wondering if there is any way to rename the field by concatenating the values of JSON attributes into existing attribute names using JOLT transformation.

Suppose we have input here:

{
  "auth_id": "0000-0000-0000",
  "read_time": "2022-01-10T00:00:00.0",
  "src_name": "REQ-A001",
  "reading_a": "150.18",
  "reading_b": "12.10",
  "reading_c": "3.00",
  "note": 1
}

What I expect is to concatenate field values from auth_id and read_time before all existing field name, with a colon (:) as separator and the result would become:

Expected:

{
  "0000-0000-0000:2022-01-10T00:00:00.0:auth_id": "0000-0000-0000-0000-0000",
  "0000-0000-0000:2022-01-10T00:00:00.0:read_time": "2022-01-10T00:00:00.0",
  "0000-0000-0000:2022-01-10T00:00:00.0:src_name": "REQ-A001",
  "0000-0000-0000:2022-01-10T00:00:00.0:reading_a": "150.18",
  "0000-0000-0000:2022-01-10T00:00:00.0:reading_b": "12.10",
  "0000-0000-0000:2022-01-10T00:00:00.0:reading_c": "3.00",
  "0000-0000-0000:2022-01-10T00:00:00.0:note": 1
}

So far I've referred to

and come up with a JOLT spec:

[
  {
    "operation": "modify-default-beta",
    "spec": {
      // concatenate field value auth_id and read_time, while using ":" as separator
      "ukey": "=concat(@(1,auth_id),':',@(1,read_time))"
    }
  },
  {
    "operation": "shift",
    "spec": {
      "*": {
        // concatenate field values before field name
        "@": "@(2,ukey):&"
      }
    }
  }
]

But I got this output:

{
  "0000-0000-0000:2022-01-10T00:00:00.0": {
    ":auth_id": "0000-0000-0000",
    ":read_time": "2022-01-10T00:00:00.0",
    ":src_name": "REQ-A001",
    ":reading_a": "150.18",
    ":reading_b": "12.10",
    ":reading_c": "3.00",
    ":note": 1
  }
}

I expect to use the JOLT spec in the JoltTransformJSON processor in Nifi.

Any help or guidance is much appreciated!

Solution

  • You can use this shorter spec:

    [
  {
    "operation": "shift",
    "spec": {
      "*": "@(1,auth_id).@(1,read_time).&"
    }
  },
  {
    "operation": "shift",
    "spec": {
      "*": {
        "*": {
          "*": "&2:&1:&"
        }
      }
    }
  }
]

    enter image description here

    Or you can use another way like below:

    [
  {
    "operation": "shift",
    "spec": {
      "*": {
        "@(2,auth_id)": "keys[#2]",
        "@(2,read_time)": "keys[#2]",
        "$": "keys[#2]",
        "@": "values"
      }
    }
  },
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "keys": {
        "*": "=join(':',@0)"
      }
    }
  },
  {
    "operation": "shift",
    "spec": {
      "values": {
        "*": "@(2,keys[#1])"
      }
    }
  }
]

    1, Shift operation Create keys and values separately.

    2, Modify operation Concate all keys with :.

    3, Shift operation Create our desired output.

    Note: Please run each spec separately to understand this code better.