How can I use source tag as index?

Question

I have a setup with Fluentd and Elasticsearch running on a Docker engine. I have swarms of services which I would like to log to Fluentd.

What I want to do is create a tag for each service that I run and use that tag as an index in Elasticsearch. Here's the setup that I have:

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match docker.service1>
  @type elasticsearch
  host "172.20.0.3"
  port 9200
  index_name service1
  type_name fluentd
  flush_interval 10s
</match>

<match docker.service2>
  @type elasticsearch
  host "172.20.0.3"
  port 9200
  index_name service2
  type_name fluentd
  flush_interval 10s
</match>

and so forth.

It would be annoying to have to include a new match tag for every single service I create, because I want to be able to add new service without updating my fluentd configuration. Is there a way to do something like this:

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match docker.**>
  @type elasticsearch
  host "172.20.0.3"
  port 9200
  index_name $(TAG)
  type_name fluentd
  flush_interval 10s
</match>

Where I use a $(TAG) variable to indicate that I want the Tag name to be the name of the index?

I've tried this from an answer I found here: ${tag_parts[0]}. This was printed literally as my index. So my index was "${tag_parts[0]}".

Thanks in advance.

Answer 1

I figured out that I needed to import the other Elasticsearch plugin. Here's an example of a match tag that I used:

<match>
   @type elasticsearch_dynamic
   host "172.20.0.3"
   port 9200
   type_name fluentd
   index_name ${tag_parts[2]}
   flush_interval 10s
   include_tag_key true
   reconnect_on_error true
</match>

I've imported the @elasticsearch_dynamic plugin instead of the @elasticsearch plugin. Then, I can use the ${tag_parts} thing.

The include_tag_key will include the tag in the json data.

It helps to read the documentation