I'm new to learning Fluent Bit, and I can't wrap my head around the benefit of specifying the Time_Key field in a parser.
An example from the documentation is below, but I don't know what the point of defining this is. By declaring Time_Key and a Time_Format it seems to take your timestamp field out of your log data and does what with it?
When I added this option to my own parser.conf and sent the logs to Splunk, my timestamp field doesn't even appear in Splunk, whereas it did before I added Time_Key. I know you can set the Time_Keep option to bring it back, but why would it remove a timestamp field from your log in the first place? Timestamp is something that seems useful to me. So, what am I missing?
From the documentation,
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S %z
The following log entry is a valid content for the parser defined above:
{"key1": 12345, "key2": "abc", "time": "2006-07-28T13:22:04Z"}
After processing, its internal representation will be:
[1154103724, {"key1"=>12345, "key2"=>"abc"}]
The time has been converted to Unix timestamp (UTC) and the map reduced to each component of the original message.
I have added the Time_Key field to my parser and have seen it remove the timestamp field from my log data. I just don't know what the purpose of doing this is?
If you don't use `Time_Key' to point to the time field in your log entry, Fluent-Bit will use the parsing time for its entry instead of the event time from the log, so the Fluent-Bit time will be different from the time in your log entry.
If you use Time_Key and Fluent-Bit detects the time, Fluent-Bit will drop the original field. You can keep it by setting Time_Keep On in your parser.conf, which will result in duplicate data.