How to making async calls to Amazon Bedrock

Question

We were trying to make calls in parallel to LLMs hosted in Bedrock, from a lambda layer (in python) only to discover that boto3 does not support async. Is there any workaround? I am looking into aiobotocore / aioboto3, but I do not find any example with Bedrock.

Any hint appreciated and thank you very much!

This is a minimal sample of the code I intended to use, but runs in sequence instead of parallel:

nest_asyncio.apply()

# async summaries
async def _into_comment(segments: list[str]):
    bedrock = boto3.client(
        service_name="bedrock-runtime",
        aws_access_key_id=aws_access_key,
        aws_secret_access_key=aws_secret_key,
        aws_session_token=aws_session_token,
        region_name=aws_region
        )
    
    async def sum_up(segment: str):
        body = json.dumps({
            "max_tokens": 256,
            "messages": [{"role": "user", "content": f"Sumarize this: {segment}"}],
            "anthropic_version": "bedrock-2023-05-31"
        })
        return bedrock.invoke_model(body=body, modelId=model_id)
    
    summaries = await asyncio.gather(*[sum_up(segment) for segment in segments])
    return summaries

summaries = asyncio.run(_into_comment(segments))

Answer 1

If you are using Anthropic, you can use the AsyncAnthropicBedrock API.

from anthropic import AsyncAnthropicBedrock

model_id = "my_model_id"
user_message = "Hello Claude!"

client = AsyncAnthropicBedrock()

message = await client.messages.create(
    model=model_id,
    max_tokens=1024,
    messages=[
        {"role": "user", "content": user_message}
    ]
)