Read file from S3 into Python memory

Question

I have a large csv file stored in S3, I would like to download, edit and reupload this file without it ever touching my hard drive, i.e. read it straight into memory from S3. I am using the python library boto3, is this possible?

Answer 1

You should look into the io module

Depending on how you want to read the file, you can create a StringIO() or BytesIO() object and download your file to this stream.

You should check out these answers:

1. How to read image file from S3 bucket directly into memory?
2. How to read a csv file from an s3 bucket using Pandas in Python