Search code examples
elasticsearchfluentddata-collection

Does fluentd meets this need

Currently I am working on a project of developing a dashboard consuming data from ElasticSearch, Elasticsearch is being fed through python scripts that gathers these information from web applications in the form of either json, data or csv files. These scripts are run via cron on a daily basis. My question here is whether I can use fluentd to replace these scripts. So can Fluentd collect information (detect if they are new or not cause some of these files are not timestamped) and output them to ElasticSearch?

Solution

  • Yes, that is exactly why Fluentd is used - to filter, parse, tags logfiles and push data to ElasticSearch.

    The python scripts that gather information from the web applications can be replaced with Fluentd and ElasticSearch can then pull this data from Fluentd.

    1. The documentation for getting json data into ElasticSearch using Fluentd is at https://docs.fluentd.org/v0.12/articles/recipe-json-to-elasticsearch.
    2. The fluent-plugin-elasticsearch plugin needed is at https://github.com/uken/fluent-plugin-elasticsearch.
    3. Documentation for the Elasticsearch Output Plugin is at https://docs.fluentd.org/v1.0/articles/out_elasticsearch.
    4. The official Fluentd documentation about how to set it all up is at https://www.fluentd.org/guides/recipes/elasticsearch-and-s3.