Search code examples
apache-nifi

NiFi - Stop upon failure

I've been trying to google and search stack for the answer but have beeen unable to find.

Using NiFi, is it possible to stop a process upon previous job failure?

We have user data we need to process but the data is sequentially constructed so that if a job fails, we need to stop further jobs from running.

I understand we can create scripts to fail a process upon previous process failure, but what if I need entire group to halt upon failure, is this possible? We don't want each job in queue to follow failure path, we want it to halt until we can look at the data and analyze the failure.

TL;DR - can we STOP a process upon a failure, not just funnel all remaining jobs into the failure flow. We want data in queues to wait until we fix, thus stop process, not just fail again and again.

Thanks for any feedback, cheers!

Edit: typos

Solution

  • You can configure backpressure on the queues to stop upstream processes. If you set the backpressure threshold to 1 on a failure queue, it would effectively stop the processor until you had a chance to address the failure.

    enter image description here

    enter image description here

    The screenshot shows failure routing back to the processor, but this is not required. It is important that the next processor should not remove it from the queue to maintain the backpressure until you take action.