Error while using confluent-kafka python library with AWS lambda

Question

I am trying to use the confluent-kafka python library to administer my cluster via a lambda function but the function fails with the error:

"Unable to import module 'Test': No module named 'confluent_kafka.cimpl'"

My requirements.txt

requests
confluent-kafka

To create the zip file I moved my code to the site-packages location of the virtual env and zipped everything.

Python Code:

import confluent_kafka.admin
import requests
def lambda_handler(event, context):
    print("Hello World")

I am using the macOS 10.X. On Linux, I noticed that pip install creates a separate confluent_kafka.libs which does not get created on mac

Answer 1

I created the required layer and can verity that it works.

The technique used includes docker tool described in the recent AWS blog:

• How do I create a Lambda layer using a simulated Lambda environment with Docker?

Thus for this question, I verified it as follows:

1. Create empty folder, e.g. mylayer.

2. Go to the folder and create requirements.txt file with the content of

echo requests > requirements.txt
echo confluent-kafka >> requirements.txt

3. Run the following docker command:

docker run -v "$PWD":/var/task "lambci/lambda:build-python3.8" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.8/site-packages/; exit"

4. Create layer as zip:

zip -r mylayer.zip python > /dev/null

5. Create lambda layer based on mylayer.zip in the AWS Console. Don't forget to specify Compatible runtimes to python3.8.

6. Test the layer in lambda using the following lambda function:

import confluent_kafka.admin
import requests

def lambda_handler(event, context):
    print(dir(confluent_kafka.admin))
    print(dir(requests))
    print("Hello World")

The function executes correctly:

['AdminClient', 'BrokerMetadata', 'CONFIG_SOURCE_DEFAULT_CONFIG', 'CONFIG_SOURCE_DYNAMIC_BROKER_CONFIG', 'CONFIG_SOURCE_DYNAMIC_DEFAULT_BROKER_CONFIG', 'CONFIG_SOURCE_DYNAMIC_TOPIC_CONFIG', 'CONFIG_SOURCE_STATIC_BROKER_CONFIG', 'CONFIG_SOURCE_UNKNOWN_CONFIG', 'ClusterMetadata', 'ConfigEntry', 'ConfigResource', 'ConfigSource', 'Enum', 'KafkaException', 'NewPartitions', 'NewTopic', 'PartitionMetadata', 'RESOURCE_ANY', 'RESOURCE_BROKER', 'RESOURCE_GROUP', 'RESOURCE_TOPIC', 'RESOURCE_UNKNOWN', 'TopicMetadata', '_AdminClientImpl', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'concurrent', 'functools']
['ConnectTimeout', 'ConnectionError', 'DependencyWarning', 'FileModeWarning', 'HTTPError', 'NullHandler', 'PreparedRequest', 'ReadTimeout', 'Request', 'RequestException', 'RequestsDependencyWarning', 'Response', 'Session', 'Timeout', 'TooManyRedirects', 'URLRequired', '__author__', '__author_email__', '__build__', '__builtins__', '__cached__', '__cake__', '__copyright__', '__description__', '__doc__', '__file__', '__license__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__title__', '__url__', '__version__', '_check_cryptography', '_internal_utils', 'adapters', 'api', 'auth', 'certs', 'chardet', 'check_compatibility', 'codes', 'compat', 'cookies', 'delete', 'exceptions', 'get', 'head', 'hooks', 'logging', 'models', 'options', 'packages', 'patch', 'post', 'put', 'request', 'session', 'sessions', 'ssl', 'status_codes', 'structures', 'urllib3', 'utils', 'warnings']
Hello World