Search code examples
fluentd

Real time data syncing using fluentd and elasticsearch


I am syncing data from kafka to elascticsearch using fluentd. But fluentd takes 60 seconds to sync data to elasticsearch. I want real time data syncing. Is there any configuration parameter which i will have to include.

i have tried

<source>
  @type kafka
  brokers localhost:9092
  topics xxx
</source>
<match xxx>
  @type elasticsearch
  scheme http
  port 9200
  <buffer tag>
    @type memory
    flush_thread_count 4
  </buffer>
</match>

Solution

  • we use flush_interval parameter, like this:

      <buffer>
        flush_interval        5s
        flush_thread_count    4
      </buffer>