elasticsearch logstash logstash-configuration

Logstash Duplicate Events

I have two configuration files for Logstash: test1.conf and test2.conf.
Each one of them has it's own flow of input -> filter -> ouput.

Both of them have the same filter and elasticsearch output writing to the same index.

My problem is that Logstash is writing duplicate events to the ElasticSearch index, no matter which input I choose to test (every event becomes two identical events instead of one).

How can I fix this?

Solution

By default, Logstash has one pipeline named main which automatically detects all .conf files in conf.d folder; this configuration is set at pipelines.yml file:

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"

If you have multiple .conf files under one pipeline, Logstash will merge them together, causing all filters and outputs to be performed on all of the inputs, so in this case, no matter which input is receiving events, it will go through two paths of filter/output, causing duplicate writing to ElasticSearch (identical events if the filters/outputs are the same for both .conf files).

Solutions

1. Move filter/output into a separate file

If your filters/outputs are the same across the config files, move filter/output into a separate file. So now you have two .conf files, one for each input, and a third .conf file for the filter/output. With this setup every input will go through only one processing path.

For example:

input1.conf

input {
  # input 1
}

input2.conf

input {
  # input 2
}

filter_output.conf

filter {
  # common filter
}  

output {
  # common output
}

You can check out this answer for another example when this solution should be chosen.

Note that If the filters/output are the same but you still want to refer them as complete different processing paths, please keep reading.

2. Split the .conf files to different pipelines

If you need every .conf file to be independent, split the .conf files to different pipelines.

In order to do that, just edit pipelines.yml file.
For example:

pipelines.yml

- pipeline.id: test1
path.config: "/etc/logstash/conf.d/test1.conf"

- pipeline.id: test2
path.config: "/etc/logstash/conf.d/test2.conf"