Reading whole S3 bucket in one request

Question

i have a bucket with 100,000 images(png\jpg) and i want to read them all in one s3 request. this is my code:

# geting connection to s3
s3 = boto3.resource('s3')
bucket = s3.Bucket(bucket_name)

# geting list of all objectsummery in the relevant dir - this is a request to the api
crops_path_list = bucket.objects.filter(Prefix=prefix)

# iterating on the list and geting objects
for index, crop in enumerate(crops_path_list):
    # getting the bytes from s3 - this a request to the api
    crop = crop.get()

    # working with the bytes to make them image
    image_content = crop['Body'].read()
    bytes_images = BytesIO(image_content)
    image = Image.open(bytes_images)
    image = image.convert("RGB")
    image = np.asarray(image)
    image = np.ascontiguousarray(image.transpose(2, 0, 1))
    image  = torch.from_numpy(np.array(image)).unsqueeze(0).to(dtype=torch.float32, device=device)

    # adding to list of all images
    images.append(image)

it takes agessss because every .get() is a request to the api. could not find any use to get the whole crops_path_list list in one request. other then running threads\subprocess\ziping, any other ideas how to use less IO and api requests?

thanks :)

Answer 1

PySpark works great for me, solved