How to get Json Data Inside cloudWatch's Log Event Using boto3

Question

I am AWS CloudWatch. I have these log events inside a log group. I can get the name, creation date etc of these log events but I wanted to get the json information inside every log events.

Suppose from the picture I wanted this output

{
    "asctime": "2019-09-20 13:55:13,604",
    "levelname": "INFO",
    "name": "root",
    "message": "Huzzah!"
}

How can do this using python? Below is my attempt. I don't think it will come in that handy.

My attempt:

import boto3
boto3.setup_default_session(region_name = "us-west-2") # e.g., 'us-eas
response = client.describe_log_groups()

#Get Group Name
logGroupName = []
for i in response['logGroups']:
    logGroupName.append(i['logGroupName'])

# Get Stream Name for every Group
logStreamName = []
for i in logGroupName:
    a = client.describe_log_streams(logGroupName=logGroupName[0],orderBy='LastEventTime')
    temp = []    
    
    for j in a['logStreams']:
        temp.append(j['logStreamName'])    
    logStreamName.append(temp)

Answer 1

You could do something like this which will iterate through the log groups and streams and add them to a nested dictionary. Regarding your question, if your logs are output in json format already then they will appear in the list associated with the log stream name.

The appropriate boto3 function was .get_log_events()

Hope this helps!

import boto3

session = boto3.session.Session()
client = session.client('logs')

response = client.describe_log_groups()

output_list = []
for resp in response['logGroups']:

    log_group_name = resp['logGroupName']
    new_entry = {log_group_name: []}

    log_streams = client.describe_log_streams(logGroupName=log_group_name, orderBy='LastEventTime')['logStreams']

    for log in log_streams:
        log_stream_name = log['logStreamName']
        out = client.get_log_events(logGroupName=log_group_name,
                                    logStreamName=log_stream_name,
                                    )['events']

        new_entry[log_group_name].append({
            log_stream_name: out
        })
    output_list.append(new_entry)

print(output_list)