Search code examples
jsonapache-nifigrok

How to grok specific fields in JSON content or grok Flowfile attribute


I have a RedPanda message queue, which NIFI pulls JSON data from. That JSON data is then reduced to only the fields I want using JOLTTransform.

The result from this is individual JSON objects.

{
  "foo": "bar",
  "message": "<Litteral copy of 1 syslog line>"
}

I want to either use a processor like ParseSyslog or ExtractGrok to parse the content of the message field and add the result from a successful parsing back into the JSON content.

So far I have been able to extract the message field into an attribute called message using EvaluateJSONPath, but this is where I get lost. I find no Grok processor or Syslog processor which can read from a specific field in the JSON content or from an attribute in the flowfile.

The question is, what can I use/do to Grok the message field and put it back into the flowfile content. I am pretty new to NIFI, so there might be obvious answers to this, but I cannot find it.

Cheers.


Solution

  • I have been using NIFI in a way, that it was not designed for. I was trying to use an axe to make NIFI do what I believed it was designed for.

    Instead of trying to work on the flowfile content, I should be routing the event to other services which do the work I want. Meaning, making NIFI a routing service instead of a "parsing" service.