Search code examples
elasticsearchoutputlogstash-configuration

How to add dynamic hosts in Elasticsearch and logstash

I have prototype working for me with Devices sending logs and then logstash parsing it and putting into elasticsearch.

Logstash output code :- 

    output{
    if [type] == "json" {
    elasticsearch {
     hosts => ["host1:9200","host2:9200","host3:9200"]
     index => "index-metrics-%{+xxxx.ww}"
    }
  }

}

Now My Question is :

I will be producing this solution. For simplicity assume that I have one Cluster and I have right now 5 nodes inside that cluster.

So I know I can give array of 5 nodes IP / Hostname in elasticsearch output plugin and then it will round robin to distribute data.

How can I avoid putting all my node IP / hostnames into logstash config file.

As system goes into production I don't want to manually go into each logstash instance and update these hosts.

What are the best practices one should follow in this case ?

My requirement is :

I want to run my ES cluster and I want to add / remove / update any number of node at any time. I need all of my logstash instances send data irrespective of changes at ES side.

Thanks.

Solution

  • If you want to add/remove/update you will need to run sed or some kind of string replacement before the service startup. Logstash configs are "compiled" and cannot be changed that way.

    hosts => [$HOSTS] ... $ HOSTS="\"host1:9200\",\"host2:9200\"" $ sed "s/\$HOSTS/$HOSTS/g" $config

    Your other option is to use environment variables for the dynamic portion, but that won't allow you to use a dynamic amount of hosts.