Fluentd - file buffer setup for redis_output (or elasticsearch) with multiprocess workers

Question

Can someone help me how to configure the file buffer for multiprocess workers in fluentd? I use this config, but when I add @type file+id to buffer for redis_store plugin, it throws this error:

failed to configure sub output copy: Plugin 'file' does not support multi workers configuration"

without id it failed with:

failed to configure sub output copy: Other 'redis_store' plugin already use same buffer path

But there is a tag in path and for different outputs (file) it works, it doesn't work only with Redis output.

I don't want to use the default memory buffer for this because of increasing memory when there is too much data. Is it possible to config this combo? (multiprocess+file buffer for redis_store plugin or Elasticsearch plugin?)

Configuration:

<system>
  workers 4
  root_dir /fluentd/log/buffer/
</system>

<worker 0-3>
<source>
 @type forward
  bind 0.0.0.0
  port 9880
</source>

<label @TEST>
<match test.**> 
 @type forest
 subtype copy
 <template>
 <store>
    @type file
    @id "file_${tag_parts[2]}/${tag_parts[3]}/${tag_parts[3]}-#{worker_id}"
    @log_level debug
    path "fluentd/log/${tag_parts[2]}/${tag_parts[3]}/${tag_parts[3]}-#{worker_id}.*.log"
    append true
 <buffer>
    flush_mode interval
    flush_interval 3
    flush_at_shutdown true
  </buffer>
 <format>
  @type single_value
  message_key log
 </format>
 </store>
 <store>
     @type redis_store
     host server_ip
     port 6379
     key test
     store_type list
 <buffer>
    #@type file CANT USE
    #id test_${tag_parts[2]}/${tag_parts[3]}/${tag_parts[3]}-#{worker_id}  WITH ID - DOESNT SUPPORT MULTIPROCESS.. 
    #path fluentd/log/${tag_parts[2]}/${tag_parts[3]}/${tag_parts[3]}-#{worker_id}.*.log WITHOUT ID - OTHER PLUGIN USE SAME BUFFER PATH
    flush_mode interval
    flush_interval 3
    flush_at_shutdown true
    flush_thread_count 4
  </buffer>
 </store>
  </template>
</match>
</label>
</worker>

Versions:

• Fluentd v1.14.3
• fluent-plugin-redis-store v0.2.0
• fluent-plugin-forest v0.3.3

Thanks!

Answer 1

The redis_store config was wrong, correct version has id under FIRST @type:

<store>
     @type redis_store
     @id test_${tag_parts[2]}/${tag_parts[3]}/${tag_parts[3]}-#{worker_id} 
     host server_ip
     port 6379
     key test
     store_type list
 <buffer>
    @type file
    flush_mode interval
    flush_interval 3
    flush_at_shutdown true
    flush_thread_count 4
  </buffer>
 </store>

Thank you for your time Azeem :)