How to send json-log to Elasticsearch using Fluentd

Question

I have Windows 10 installed on my computer and Elasticsearch/Kibana running in docker container.

I am trying to redirect logs generated by my application to Elasticsearch using Fluentd. Here is td-agent.conf file:

<source>
  @type tail
  path C:/Projects/log.json
  pos_file C:/Projects/log.json.pos
  tag *
  format json
  time_key @timestamp
</source>

<match **>
  @type elasticsearch
  logstash_format false
  host localhost
  port 9200
  index_name appname-api-*
  type_name fluentd
  flush_interval 1s
</match>

Here is part from appsettings where I specified that I want json-file in elasticsearch format:

  {
    "Name": "File",
    "Args": {
      "path": "c:/Projects/log.json",
      "formatter": "Serilog.Formatting.Elasticsearch.ElasticsearchJsonFormatter, Serilog.Formatting.Elasticsearch"
    }
  }

Here is a line from log file:

{"@timestamp":"2021-10-22T11:13:39.4325643+03:00","level":"Information","messageTemplate":"Now listening on: {address}","message":"Now listening on: \"http://localhost:5001\"","fields":{"address":"http://localhost:5001","SourceContext":"Microsoft.Hosting.Lifetime","MachineName":"MACHINENAME"}}

But it is not working. I suspect my td-agent.conf. Could you please provide me with some example?

Or maybe it is easier to switch to Filebeat or something else?

Answer 1

Got it! Here is correct td-agent.conf

<system>
  log_level debug
</system>

<source>
  @type tail
  path C:/Projects/log.json
  pos_file C:/Projects/log.json.pos
  tag log_test
  emit_unmatched_lines true

  <parse>
    @type json
  </parse>
</source>

<match log_test>
  @type elasticsearch
  host localhost
  port 9200
  index_name appname-api-2021-10
  type_name _doc
  flush_interval 1s
</match>