How to use includes in logstash conf files?

Question

Can includes be used in logstash config files?

Minimal, Complete, and Verifiable Example

Can I replace this...

file: beats.conf

input {
  beats {
    port => 5044
  }
}
filter {
    date {
        match => ["myTimestamp", "yyyyMMdd_HH:mm:ss.SSS"]
        target => "date_time"
    }
}
output {
  elasticsearch {
    hosts => [ "localhost:9200" ]
  }
}

...with this?

file: date.inc

date {
    match => ["myTimestamp", "yyyyMMdd_HH:mm:ss.SSS"]
    target => "date_time"
}

file: beats.conf

input {
  beats {
    port => 5044
  }
}
filter {
    #include <date.inc>  // <- THIS THIS THIS THIS THIS
}
output {
  elasticsearch {
    hosts => [ "localhost:9200" ]
  }
}

Answer 1

Actually there is no support for "include" and Logstash is not able to load a pipeline splitted in different files in order to reuse common parts. EDIT: The only way to compose a pipeline from different file is to specify a folder or wildcard "*" in the path.config setting so that config files are read in alphabetical order (Thanks to @Badger).

If you don't want to define your own pipeline's composition/compilation system, you could take a look on "Pipeline-to-Pipeline" communication that can be used for example to break-up you complex pipelines and reuse your filters on different flows: https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html. Note that with this approach you will pay the overhead of running multiple pipelines.

For example:

pipelines.yml

- pipeline.id: input
  path.config: "<path-to-file>/beats.conf"
- pipeline.id: date-filters
  # This common pipeline allow to reuse the same logic for complex filters
  path.config: "<path-to-file>/date.conf"
- pipeline.id: output
  path.config: "<path-to-file>/elasticsearch.conf"

beats.conf

input {
  beats {
    port => 5044
  }
}
output { pipeline { send_to => [commonFilters] } }

date.conf

input { pipeline { address => commonFilters } }
filter {
  date {
    match => ["myTimestamp", "yyyyMMdd_HH:mm:ss.SSS"]
    target => "date_time"
  }
}
output { pipeline { send_to => [output] } }

elasticsearch.conf

input { pipeline { address => output } }
output {
  elasticsearch {
    hosts => [ "localhost:9200" ]
  }
}