Search code examples
amazon-s3boto3

Connection pool is full warning while reading S3 objects via multiple threads

I am using boto3 to read s3 objects

s3_client = boto3.client('s3', region_name='us-east-1')
obj = s3_client.get_object(Bucket=S3_BUCKET, Key=key)

I am running this via 50-100 threads to access different objects and getting warning :

urllib3.connectionpool - WARNING - Connection pool is full, discarding connection: s3.amazonaws.com

How can I increase the connection pool size?

Is there any better way to access different S3 objects with multiple threads?

Solution

  • Adding max_pool_connections (default 10) solved it.

    client_config = botocore.config.Config(
    max_pool_connections=50
)
s3_client = boto3.client('s3', region_name='us-east-1', config=client_config)