Logs are ignoring input section in config files

Question

I have a simple setup for capturing logs though HTTP and TCP. I've created 2 conf files at /etc/logstash/conf.d/ (see below) but logs sent though HTTP are also being passed through the TCP pipeline and vise versa. For example when I send a log through TCP it ends up both in http-logger-* index and in tcp-logger-*.. it makes no sense to me :(

http_logger.conf

input {
  http {
    port => 9884
  }
}
filter {
    grok {
      match => ["[headers][request_path]", "\/(?<component>[\w-]*)(?:\/)?(?<env>[\w-]*)(?:\/)?"]
    }
}
output {
    amazon_es {
        hosts => ['XXXXX']
        region => 'us-west-2'
        aws_access_key_id => 'XXXXX'
        aws_secret_access_key => 'XXXXX'
        index => 'http-logger-%{+YYYY.MM.dd}'
    }
    stdout { codec => rubydebug }
}

tcp_logger.conf

input {
  tcp {
    port => 9885
    codec => json
  }
}
filter {

}
output {
    amazon_es {
        hosts => ['XXXXX']
        region => 'us-west-2'
        aws_access_key_id => 'XXXXX'
        aws_secret_access_key => 'XXXXX'
        index => 'tcp-logger-%{+YYYY.MM.dd}'
    }
    stdout { codec => rubydebug }
}

Any ideas on what am I missing? Thank you

Answer 1

The explanation provided by @Ram is spot on however there is a cleaner way of solving the issue: enter pipelines.yml.

By default it looks like this:

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"

basically it loads and combines all *.conf files - in my case I had two.

To solve the issue just separate the pipelines like so:

- pipeline.id: httplogger
  path.config: "/etc/logstash/conf.d/http_logger.conf"
- pipeline.id: tcplogger
  path.config: "/etc/logstash/conf.d/tcp_logger.conf"

The pipelines are now running separately :)

P.S. Don't forget to reload logstash after any changes here