I would like to use the current day as timestamp (date) as this information isn't available in our logfiles. Example -> main_core.log:
04:00:19.675 [ActiveMQ Task-9] INFO a.b.c.t.failover.FailoverTransport - Successfully reconnected to ssl://localhost:12345
And i would like to split it and use the current day as date and the timestamp out of the logfile.
Is that even possible?
Thanks and many regards
You can add a field with the part of the timestamp that is missing in your log and then concatenate with a variable that contains the hour and use it as your @timestamp field.
The filter below does something like this:
filter {
grok {
break_on_match => false
match => ["message","%{TIME:hour} %{GREEDYDATA:msg}"]
tag_on_failure => [ "_grokparsefailure"]
add_field => { "time" => "%{+YYYY-MM-dd}"}
add_field => { "timestamp" => "%{time} %{hour}" }
}
date {
target => "@timestamp"
match => ["timestamp", "YYYY-MM-dd HH:mm:ss.SSS"]
}
}
First it will match your message with a grok pattern that will extract the hour and save it in a field name hour
, and the rest will be saved in a field name msg
, but you can parse the rest if you want.
Then it will add a field name time
with the pattern YYYY-MM-dd, for example 2018-07-12
.
After that, it will create a field named timestamp
the field time
with the field hour
, which will result in 2018-07-12 4:00:19.675
The date
filter is used to use your generated timestamp as the default timestamp field in elastic, which is @timestamp
.
A logstash output for this filter is something like this:
{
"@timestamp":"2018-07-12T04:00:19.675Z",
"message":"04:00:19.675 [ActiveMQ Task-9] INFO a.b.c.t.failover.FailoverTransport - Successfully reconnected to ssl://localhost:12345",
"timestamp":"2018-07-12 04:00:19.675",
"msg":"[ActiveMQ Task-9] INFO a.b.c.t.failover.FailoverTransport - Successfully reconnected to ssl://localhost:12345",
"time":"2018-07-12",
"@version":"1",
"hour":"04:00:19.675",
"host":"logstash-hostname"
}