configuring AWS Network Load Balancer for logstash cluster Running in ASG

Question

• My Aim is to setup A common logstashcluster that scales, as I am
  using filebeat in my backup servers for collection of logs .

• For that I setup an ELB that points out to the Logstash cluster (Auto scaling group)

• In ELB target group settings i gave the health check port as 9600, as my logstash is running at 9600 port.

• But when The ELB runs the healthcheck it says always unhealthy instance

• I am not sure what i am doing wrong here
• My logstash config

input {
 beats {
   type => "testlog"
   port => "5066"
 }
}
filter {
   if [message] =~ /{.*}/ {
       grok { match => { "message" => "(?<[@metadata][json]>({.*}))"} }
       json { source => "[@metadata][json]" remove_field => [ "message" ] }
   }
}

output {
 stdout {
       codec => rubydebug
       }
 amazon_es {
   hosts => ["****************************"]
   region => "us-east-1"
   index => "filebeatsecondpipelinefinal1-%{+YYYY.MM.dd}"
   #user => "elastic"
   #password => "changeme"
 }
}

• Im having multiple pipeline setup so another config uses this

input {
  beats {
    type => "testlog"
    port => "5044"
  }
}
filter {
    if [message] =~ /{.*}/ {
        grok { match => { "message" => "(?<[@metadata][json]>({.*}))"} }
        json { source => "[@metadata][json]" remove_field => [ "message" ] }
    }
}

output {
  stdout {
        codec => rubydebug
        }
  amazon_es {
    hosts => ["************************"]
    region => "us-east-1"
    index => "filebeatsecondpipelinefinal2-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
}

Answer 1

You need to make some changes to your ELB and Logstash configuration.

First, the port 9600 is the REST port to get logstash metrics, which you can use to do healthcecks, but by default and per security reasons, logstash binds this port to the loopback ip (127.0.0.1), you will need to add the http.host config in your logstash.yml to bind it to the internal IP of the intance.

http.host: "instance-local-ip"

You need to do that in every logstash host, you can also use environment variables in the logstash config.

Second, your ELB target group is using the wrong port. Your pipelines are using the ports 5044 and 5066, so you will need a target group for port 5044 and another one for port 5066, and when configuring the healthcheck for those target group you will need to chose the override port option and use the port 9600.

This way your target group will listen on the port 5044 or 5066 but will perform the healthcheck for your instances on port 9600.