Permission Check Successful in the process of moving the CloudWatch log to S3

Question

How can I save the logs of CloudWatch as S3?

This is the S3 bucket policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.ap-northeast-2.amazonaws.com"
            },
            "Action": "s3:GetBucketAcl",
            "Resource": "arn:aws:s3:::s3-bucket-logs"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.ap-northeast-2.amazonaws.com"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::s3-bucket-logs/*",
            "Condition": {
                "StringEquals": {
                    "s3:x-amz-acl": "bucket-owner-full-control"
                }
            }
        }
    ]
}

Cron event was generated by EventBridge.

Just I get Permission Check Successful.

This is code and already set environmental variables.

import boto3
import os
import datetime
import time

DESTINATION_BUCKET = os.environ['DESTINATION_BUCKET']
GROUP_NAME = os.environ['GROUP_NAME']
PREFIX = os.environ['PREFIX']
BATCH_SIZE = int(os.environ['BATCH_SIZE'])
DAYS = int(os.environ['DAYS'])
EXPORT_CHECK_CYCLE_TIME = int(os.environ['EXPORT_CHECK_CYCLE_TIME'])

currentTime = datetime.datetime.now()
start_date = currentTime - datetime.timedelta(days=(DAYS * 2))
end_date = currentTime - datetime.timedelta(days=DAYS)

FROM_DATE = int(start_date.timestamp() * 1000)
TO_DATE = int(end_date.timestamp() * 1000)

#S3 Log Bucket Directory
BUCKET_PREFIX = os.path.join(PREFIX, end_date.strftime('%Y{0}%m{0}%d').format(os.path.sep))

#1. Export CloudWatch logs in logGroup within a specified time range to S3
#2. Wait for the export operation to finish(because create_export_task is an asynchronous call)
#3. Check CloudWatch LogStream whether log events that need to be preserved exists or not
#4. If there are logs to be preserved, store logs in variables
#5. delete & remake LogStream with the same name as previously used and put the archived logs in LogStream

def lambda_handler(event, context):
    client = boto3.client('logs')
    #Export CloudWatch Log To S3
    response = client.create_export_task(
         logGroupName=GROUP_NAME,
         fromTime=FROM_DATE,
         to=TO_DATE,
         destination=DESTINATION_BUCKET,
         destinationPrefix=BUCKET_PREFIX
        )
    #Check create_export_task is finished
    taskId = response['taskId']
    status = 'RUNNING'
    while status in ['RUNNING','PENDING']:
        time.sleep(EXPORT_CHECK_CYCLE_TIME)
        response_desc = client.describe_export_tasks(
            taskId=taskId
        )
        status = response_desc['exportTasks'][0]['status']['code']
    #If create_export_task is finished 
    if status == 'COMPLETED':
        #Get all LogStreams in LogGroup
        log_streams = client.describe_log_streams(logGroupName=GROUP_NAME)['logStreams']
        for stream in log_streams:
            stream_name = stream['logStreamName']
            #If you have reached the end of the stream, it returns the same token you passed in
            prev_token = 'prev_token'
            next_token = None
            kwargs = dict(
                    logGroupName=GROUP_NAME,
                    logStreamName=stream_name,
                    startTime=TO_DATE,
                    limit=BATCH_SIZE,
                    startFromHead=False
                )
            retention_events = []
            while next_token != prev_token:
                #Get batch size LogEvents in LogStream at a time, in order of latest
                if next_token is not None:
                    kwargs['nextToken'] = next_token
                log_event_info = client.get_log_events(**kwargs)
                events = log_event_info['events']
                prev_token = next_token
                next_token = log_event_info['nextForwardToken']
                for event in events:
                    if event['timestamp'] <= TO_DATE:
                        break
                    #Remove keys not needed in put_log_events function
                    del event['ingestionTime']
                    retention_events.append(event)
            #Delete & remake LogStream
            client.delete_log_stream(logGroupName=GROUP_NAME, logStreamName=stream_name)
            client.create_log_stream(logGroupName=GROUP_NAME, logStreamName=stream_name)
            #If there are log events that need to be preserved in LogStream, Put batch size LogEvents in LogStream at a time
            retention_events_size = len(retention_events)
            for i in range(0, retention_events_size, BATCH_SIZE):
                client.put_log_events(
                    logGroupName=GROUP_NAME,
                    logStreamName=stream_name,
                    logEvents=retention_events[i : (i + BATCH_SIZE)]
                )
                time.sleep(0.2)

This is environment

• BATCH_SIZE = 32
• DAYS = 1
• DESTINATION_BUCKET = s3-bucket-logs
• EXPORT_CHECK_CYCLE_TIME = 3
• GROUP_NAME = logs
• PREFIX = save

lambda test error [ERROR] InvalidParameterException: An error occurred (InvalidParameterException) when calling the CreateExportTask operation: PutObject call on the given bucket failed. Please check if CloudWatch Logs has been granted permission to perform this operation. This may be a result of a KMS key or bucket encryption misconfiguration.

A file with name 'aws-log-write-test' gets created on S3 inside the bucket, but there is no other data or file in the bucket. How can I solve?

Answer 1

My account was encrypted and was supposed to be transferred to an encrypted bucket, so I followed option 4 steps to fulfill it.

It worked successfully.

reference

https://docs.aws.amazon.com/ko_kr/AmazonCloudWatch/latest/logs/S3ExportTasksConsole.html