Search code examples
apache-nifi

How to get NiFi processor error into Flowfile attribute?

I have a PutGCSObject processor for which I want to capture the error into a flow file attribute.

enter image description here

As in the Picture, when there is an error for the Processor, it sends to failure with all the pre-existing attributes as-is.

enter image description here

I want the error message to be a part of the same flow file as an attribute. How can I achieve that ?

Solution

  • There is actually a way to get it.

    Here is how i do it:

    1: I route all ERROR connections to a main "monitoring process group"

    2: Here is my "monitoring process group"

    enter image description here

    In updateattribute I capture filename as initial_filename

    Then in my next step I query the bulletins

    enter image description here

    I then parse the output as individual attributes.

    After I have the parsed bulleting output I use a RouteOnAttribute proc to drop all bulletins I don't need (some of them I have already used and notified on).

    enter image description here

    Once I only have my actual ERROR bulletin left, I use ExecuteStreamingCommand to run a python script using nipyapi module to get more info about the error, such as where it is in my flow, hierarchy, a description of the processor that failed, some proc stats and also I have metadata catalog about each proc/process group with their custodians and business use case.

    This data is then posted to sumologic for logging and also I trigger a series of notifications (Slack + PagerDuty hook to create an incident lifecycle).

    I hope this helps