Is there a way to feed AWS Cloudwatch logs into a Kafka Topic

Question

I'm struggling to find any direction on this. I have an internal system that will process the logs for monitoring. I want errors to be sent from Cloudwatch and published on a kafka topic from which my monitoring system will read from.

Does Cloudwatch support this setup, if so how? Is this the right architectural approach?

I'm early in my AWS journey.

Answer 1

To feed AWS CloudWatch Logs into a Kafka Topic, you can indeed leverage Amazon Kinesis Data Firehose by creating a subscription filter in CloudWatch Logs. This method allows for the efficient transfer and transformation of log data into a Kafka cluster.

Here's a refined overview of the process:

1. CloudWatch Logs: First, ensure that your logs are being collected in CloudWatch Logs. This is a prerequisite for setting up the log subscription filters.

2. Log Subscription Filter: Utilize the log subscription filter feature in CloudWatch Logs. This allows you to deliver real-time log events from multiple or all log groups in your account to services such as Amazon Kinesis Data Firehose. For more details, refer to the AWS Logs Subscription Filter documentation.

3. Amazon Kinesis Data Firehose: Set up a Kinesis Data Firehose delivery stream, designating your Kafka cluster as the destination. During this setup, you have the option to configure the stream for any necessary transformations of the incoming log data before it gets forwarded to your Kafka topic.

These steps provide a streamlined method to forward logs from CloudWatch to Kinesis Data Firehose, and subsequently into a Kafka cluster.

References/Documentations:

• Writing to Kinesis Data Firehose Using CloudWatch Logs documentation
• AWS Logs Subscription Filter