Search code examples
elasticsearchlogstashkibanafluentd

Fluentd High Availability Custom Index

I've setup a fluentd/elasticsearch/kibana stack very similar to what is described here. When I look at the logs in kibana I notice that they are automatically indexed by day using the format "logstash-[YYYY].[MM].[DD]. Based on the documentation for the fluentd elasticsearch plugin it seems that you can create a custom index by setting the "index_name" property.

I've tried this on both the log forwarder and the log aggregator but I still seem to get the default index name in elasticsearch. Is there something else required to customize this index name in a HA setup?

Here is the log forwarder config:

<source>
  type tail
  path /var/log/debug.json
  pos_file /var/log/debug.pos
  tag asdf
  format json
  index_name fluentd
  time_key time_field
</source>

<match *>
  type copy
  <store>
    type stdout
  </store>
  <store>
    type forward
    flush_interval 10s
    <server>
      host [fluentd aggregator]
    </server>
  </store>
</match>

And here is the log aggregator config:

<source>
  type forward
  port 24224
  bind 0.0.0.0
</source>

<match *>
  type copy
  <store>
    type stdout
  </store>
  <store>
    type elasticsearch
    host localhost
    port 9200
    index_name fluentd
    type_name fluentd
    logstash_format true
    include_tag_key true
    flush_interval 10s # for testing
  </store>
</match>
Solution

  • I found an issue on the fluent-plugin-elasticsearch repo that explains this behavior. When setting the "logstash_format" option to true the "index_name" field is ignored.