Elastic Pipelines: Skip Import On Failure

Question

Individual processors in an Elastic pipelines have an on_failure attribute. This allows you to handle a failure/error in a pipeline. The example in the docs show setting an additional field on your document.

{
  "description" : "my first pipeline with handled exceptions",
  "processors" : [
    {
      "rename" : {
        "field" : "foo",
        "to" : "bar",
        "on_failure" : [
          {
            "set" : {
              "field" : "error.message",
              "value" : "{{ _ingest.on_failure_message }}"
            }
          }
        ]
      }
    }
  ]
}

Is it possible to tell the pipeline to SKIP importing a document if any processors in the pipeline fail?

Answer 1

You can "hijack" the drop processor to skip either directly in the on_failure step (no need for _ingest.on_failure_message if you're aborting anyways):

{
  "description": "my first pipeline with handled exceptions",
  "processors": [
    {
      "rename" : {
        "field" : "foo",
        "target_field": "bar",
        "on_failure" : [
          {
            "drop" : {
              "if" : "true"
            }
          }
        ]
      }
    }
  ]
}

or use it as a separate processor, perhaps at the very end, after ctx.error has been set by any of the processors' on_failure handlers:

{
  "description": "my first pipeline with handled exceptions",
  "processors": [
    {
      "rename" : {
        "field" : "foo",
        "to" : "bar",
        "on_failure" : [
          {
            "set" : {
              "field" : "error.message",
              "value" : "{{ _ingest.on_failure_message }}"
            }
          }
        ]
      }
    },
    {
      "drop": {
        "if": "ctx.error.size() != null"
      }
    }
  ]
}

Both of these will result in a noop when the pipeline is applied.