AWS Lambda Boto3 DynamoDB Concurrency

Question

I am trying to use concurrent.futures to retrieve items from AWS DynamoDB asynchronously to shorten the function time.

def retrieve_dynamo_item(dynamo_tablename, key):
    dynamodb = boto3.resource('dynamodb')
    dynamo_table = dynamodb.Table(dynamo_tablename)
    dynamo_result = dynamo_table.get_item(Key=key)
    return dynamo_result.get('Item')

def lambda_handler(event, context):

    dynamo_queries = ## code
    with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
        futures = [executor.submit(retrieve_dynamo_item, dynamo_table, key) for dynamo_table, key in dynamo_queries]

    final_result = []
    for f in concurrent.futures.as_completed(futures):
        if f.result():
            final_result += f.result().get('result')

However when deploying and testing the lambda function performance does not seem to be improved but in fact worse than just running the queries synchronously. Any suggestions on what could be the reason?

Answer 1

As Oleksii already said, the default memory setting of Lambda will only give you 1 CPU. So concurrency is kind of hard to achieve. The slower performance might be due to the added overhead of handling threads etc.

The following table contains the mapping of memory to CPU cores at this moment (could change in the future).

Memory	CPU cores
128 - 1769	1
1770 - 3538	2
3539 - 5307	3
5308 - 7076	4
7077 - 8845	5
8846 - 10240	6

So you could increase your Lambdas memory to 2000MB for example and check if that improves performance.

Furthermore, I'd definitely recommend using Batch operations instead of just GetItem.