How to set retention days to a logstream with cloudformation or yaml

Question

So, I have been working back and forth with this task.

I have AWS CloudWatch logs where I have to apply retention policies to some logstreams of a Loggroup

Loggroup: "My Loggroup" and inside it I have different logstreams Logstream-a Logstream-b

what I want to do, is to set different retention days to each logstream.

Logstream-a = 180 days Logstream-b = 90 days

I've found that I can set the retention settings to a loggroup in YAML, but I can't do that to a logstream.

CloudWatchLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: MyLoggroup
      RetentionInDays: 180

Is there any way to apply retention settings to a LogStream with Python, Cloudformation or YAML?

Answer 1

As you already discovered, you can set log retention on LogGroups, but not on LogStreams.

You could use CloudWatch S3 exports to differentiate retention by LogStream indirectly. What makes S3 exports work for your use case is that export tasks can filter exports by logStreamNamePrefix.

# CloudWatchLogs.Client
response = client.create_export_task(
    taskName='string',
    logGroupName='string',
    logStreamNamePrefix='string', # Export only log streams that match the provided prefix
    fromTime=123,
    to=123,
    destination='string',
    destinationPrefix='string'
)

• Set the LogGroup retention days to the shortest retention category. All logs in CW expire at that time.
• Create export tasks for the LogSteams that belong to the longer retention categories.
• Create expiry lifecycle rules in S3 to automatically delete objects after a set number of days. Rules can filter by object prefix, which handily corresponds to the log stream names.
• Run the export tasks periodically.