AWS Cloud Watch: How to specify which field to use for timestamp in json?

Question

I have

datetime_format = "%Y-%m-%dT%H:%M:%S.%f%z"

in /etc/awslogs/awslogs.conf

And I have log like this:

{
    "level": "info",
    "ts": "2023-01-08T21:46:03.381067Z",
    "caller": "bot/bot.go:172",
    "msg": "Creating test subscription declined",
    "user_id": "0394c017-2a94-416c-940c-31b1aadb12ee"
}

However timestamp does not parsed

I see warning in logs

2023-01-08 21:46:03,423 - cwlogs.push.reader - WARNING - 9500 - Thread-4 - Fall back to previous event time: {'timestamp': 1673211877689, 'start_position': 6469L, 'end_position': 6640L}, previousEventTime: 1673211877689, reason: timestamp could not be parsed from message.

upd:

tried to remove level

{
    "ts": "2023-01-08T23:15:00.518545Z",
    "caller": "bot/bot.go:172",
    "msg": "Creating test subscription declined",
    "user_id": "0394c017-2a94-416c-940c-31b1aadb12ee"
}

and still does not work.

Answer 1

There 2 different formats of cloudwatch log configurations:

1. https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html. This is deprecated as mentioned in the alert section of the page.
2. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html. This is the configuration for new unified cloudwatch agent and it doesn't have the parameter datetime_format to configure. Instead it has the timestamp_format.

Since you have mentioned the datetime_format, I'm assuming you are using the old agent. In that case, the %z refers to UTC offset in the form +HHMM or -HHMM. +0000, -0400, +1030 as per the linked documentation[1 above]. Your timestamp doesn't have an offset mentioned hence your format should be %Y-%m-%dT%H:%M:%S.%fZ. There the Z is similar to T where it just represents a character. Also, specify the time_zone as UTC.