Can I consume an AWS MSK kafka topic from a Lambda function?

Question

I thought it would be trivial to use a lambda to consume messages coming to a topic in a AWS MSK Kafka cluster, but I could not find a way to do it from the AWS docs. Is it possible in some way?

Answer 1

Yes. You can by using a Kafka client library in you Lambda code. Find an example for Python below:

#!/bin/env python

import json
import logging
import time
import os
from kafka import KafkaConsumer               


"""
Read data from MSK and console it out.

Required environment variables:
MSK_BOOTSTRAP_SRV: MSK Bootstrap servers. 
MSK_TOPIC_NAME: MSK topic. 

"""
kafka_client = None
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)

def lambda_handler(event, context):
    logger.debug('MSK consumer starts.')
    try:
        consumer = KafkaConsumer(os.environ['MSK_TOPIC_NAME'], auto_offset_reset='earliest',
                             bootstrap_servers=os.environ['MSK_BOOTSTRAP_SRV'], api_version=(0, 10), consumer_timeout_ms=1000)
        
        for msg in consumer:
            print(msg.value)

        if consumer is not None:
            consumer.close()
            
    except Exception as ex:
        logger.error('Exception: {}'.format(ex))
    return