ELK - Duplicate log

Question

I have a weird problem with Logstash (v7.11) and Elasticsearch.

I currently have two configuration files:

• 01-beats-syslog.conf (with logs sent by winlogbeat)

   input {
    beats {
      port => 5044
      ssl => false
     }
   }
  
   filter {
   if [type] == "syslog" {
       grok {
         match => { "message" => "%{SYSLOGLINE}" }
       }
  
       date {
   match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
   }
     }
  
   }
  
   output {
    elasticsearch {
     hosts => localhost
       index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
          }
   stdout {
       codec => rubydebug
          }
   }
  

• 02-network-syslog.conf (with logs of agentless devices (e.g. switches, firewalls, etc.))

   input {
     tcp {
       port => 514
       type => syslog
     }
     udp {
       port => 514
       type => syslog
     }
   }
  
   filter {
     if [type] == "syslog" {
       grok {
         match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
         add_field => [ "received_at", "%{@timestamp}" ]
         add_field => [ "received_from", "%{host}" ]
       }
       date {
         match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
       }
     }
   }
  
   output {
    elasticsearch {
       hosts => ["localhost:9200"]
       index => "syslog-%{+YYYY.MM}"
          }
   stdout {
       codec => rubydebug
          }
   }
  

for some reason, a copy of the "winlogbeat" logs end up inside the "network-syslog" index (in addition to the true syslog traffic).. each configuration file is listening on a different port, what's wrong with the configuration?

I also checked that I don't have firewall rules that forward traffic to 514, in fact with tcpdump I don't see traffic coming from winlogbeat on that port.

Answer 1

If you point path.config to a directory then logstash will concatenate all the files in the directory, read events from all the inputs, run them through all the filters, and send all of them to all the outputs. Either configure pipelines.yml to run each configuration file in a separate pipeline or use a conditional around the output

 output {
   if [type] == "syslog" {
      elasticsearch {
        hosts => ["localhost:9200"] ...