I'm currently reading container logs as my source in fluentd, and I'm parsing all of our log files which is in JSON format. The JSON parser is working as expected based on our configuration, but the issue is the time format. There are multiple time formats depending on the log file.
As an example, one log file might store time as 2020-05-28T21:00:07Z, but another will store it as 2020-04-28T20:07:52.013557931Z. The time is stored in a key named "time" in all cases, too, so fluentd by default attempts to parse this key.
Is it possible to use the same filter to parse all JSON data but have it handle multiple time formats? Would we need to create different filters for each different time format?
This is more of a 'fluentd' question, than a straight 'Ruby' question.
However, to your specific question about filters, generally speaking it's impossible to tell specific date formats based on the value alone. For example, does '2020-03-04' mean March 4 (yyyy-MM-dd), or April 3 (yyyy-dd-MM)? Unfortunately there's no way to tell from values alone.
The best course of action is generally to verify the format per source (by reading documentation for that source or otherwise), and make sure identifying information is available to tie the date value back to the source. Without that, neither fluentd nor any other program will be able to determine what time it is (although you can do tricks in some situations to determine what time it isn't.)