Search code examples
elasticsearchkuberneteselastic-stackfilebeatelasticsearch-painless

Using Elasticsearch and filebeats how do I only execute my pipeline on certain files?

I only want to run my pipeline on files where the log path contains a certain keyword, how do I do this within the pipeline?

Pipeline (removed my pattern and patterns as it is not relevant):

{
  "description" : "...",
  "processors": [
    {
      "grok": {
        "if": "ctx['log']['file']['path'].value.contains('keyword')",
        "field": "message",
      }
    }
  ]
}

In Kibana I see I have log.file.path available as metadata, and I just want to run the pipeline if it contains a keyword, but I get a runtime error because of my if statement.

Thanks for your help!

EDIT: I think the problem lies with how I am trying to access the log.file.path field as I don't know how to reference it correctly from here.

Solution

  • You can probably use the Drop processor https://www.elastic.co/guide/en/elasticsearch/reference/current/drop-processor.html

    "drop": {
   "if": "ctx.log.file.path.contains('keyword');"
}

    You can find more complexe exemples here: https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest-conditional-complex.html