Search code examples
elasticsearchlogstashlogstash-configurationfilebeatmetricbeat

how to override timestamp field coming from json in logstash

I have the following below kind of json present with me to be dumped in elastic search using filebeat {"@timestamp":"2017-02-10T06:30:51.424Z","beat":{"hostname":"myhostname","name":"mydevice-logger","version":"5.2.0"},"fields":{"device_type":"mydevice","env":"staging"},"metricset":{"module":"system","name":"cpu","rtt":211},"system":{"cpu":{"cores":4,"idle":{"pct":0.000000},"iowait":{"pct":0.000000},"irq":{"pct":0.000000},"nice":{"pct":0.000000},"softirq":{"pct":0.000000},"steal":{"pct":0.000000},"system":{"pct":0.000000},"user":{"pct":0.000000}}},"tags":["automata","box"],"type":"metricbeat-test-log"}

my logstash( version 5.1.1) config contains, input, filter and output like below - 

input { 
  beats {
        port => 5046
        codec => json
  }
}

filter {
    if ...{}
    else if [type] == "metricbeat-test-log" {

      date {
        match => ["@timestamp", "ISO8601"]
      }

      }
    }

}


output {
    if ...{}
    else if [type] == "metricbeat-test-log" {
        stdout { codec => rubydebug   }
    }
}

The type is right however the date filter is not working . The @timestamp finally takes current timestamp always . I want to replace it with original @timestamp present in json.

Solution

  • I have got the answer following this old thread https://discuss.elastic.co/t/replace-timestamp-with-actual-timestamp-from-log-file/72017 which explains that the "@timestamp":"2017-02-10T06:30:51.424Z" in log line above is a JSON representation of the @timestamp field and its value. Following the suggestions I added the json configuration in filebeat and it worked for me . 

    - input_type: log
  paths:
    - /var/logs/mylogs/*.log
  fields:
    environment: testing
  document_type: test-metric-document

  json.keys_under_root: true   # added this line 
  json.overwrite_keys: true    # added this line

    Though I am not happy with this solution as my real need was to get the both timestamp, the logstash event one(in some other variable ) and the timestamp from log in @timestamp.