How to parse JSON data from syslog with fluentd?

Question

My custom rsyslog template:

template(name="outfmt" type="list" option.jsonf="on") {
         property(outname="@timestamp"
                  name="timereported" 
                  dateFormat="rfc3339" format="jsonf")
         property(outname="host"
                  name="hostname" format="jsonf")
         property(outname="severity"
                  name="syslogseverity-text" caseConversion="upper" format="jsonf")
         property(outname="facility"
                  name="syslogfacility-text" format="jsonf")
         property(outname="syslog-tag"
                  name="syslogtag" format="jsonf")
         property(outname="source"
                  name="app-name" format="jsonf")
         property(outname="message"
                  name="msg" format="jsonf")

     }

My rsyslog example output:

{
  "@timestamp": "2018-03-01T01:00:00+00:00",
  "host": "172.20.245.8",
  "severity": "DEBUG",
  "facility": "local4",
  "syslog-tag": "app[1666]",
  "source": "app",
  "message": " this is my syslog message"
}

How can I parse this log with fluentd and send to elasticsearch?

Answer 1

You can receive logs directly in elasticsearch (without even having to format them to json) through the syslog plugin. This probably would be the most straightforward solution to your problem.

If for some reason u need to use some kind of log aggregator, I personally would not recommend fluentd, as it can bring unecessary complexity with it.

But you could use logstash which is supported by elasticsearch and you can find plenty of documentation about it.