In Memory Zip File Python Error

Question

I'm trying to make an in-memory zip file in Python and upload it to Amazon S3. I've read the similar posts on the matter, but no matter what I try, Windows and Linux (RHEL5) cannot open it (it's corrupt). Here's the code I'm running:

f_redirects = StringIO()
f_links = StringIO()
f_metadata = StringIO()

# Write to the "files"

zip_file = StringIO()
zip = zipfile.ZipFile(zip_file, 'a', zipfile.ZIP_DEFLATED, False)
zip.writestr('redirects.csv', f_redirects.getvalue())
zip.writestr('links.csv', f_bad_links.getvalue())
zip.writestr('metadata.csv', f_metadata.getvalue())

f_redirects.close()
f_links.close()
f_metadata.close()

k = Key(BUCKET)
k.key = '%s.zip' % base_name
k.set_metadata('Content-Type', 'application/zip')
k.set_contents_from_string(zip_file.getvalue())
zip.close()
zip_file.close()

Answer 1

The problem is that you're trying to use the contents of the ZipFile before you call close on it.

As the documentation says:

You must call close() … or essential records will not be written.

On top of that, although it sometimes works, it's actually not legal to call getvalue() on a closed StringIO. Again, from the docs:

Return a str containing the entire contents of the buffer at any time before the StringIO object’s close() method is called.

Finally, if you're using Python 3.x, you probably want to use BytesIO rather than StringIO. In fact, you might want to use BytesIO even in 2.x, as long as you're using 2.6+.

---

Also, your code would be a lot more readable, and harder to get wrong, if you used with statements instead of trying to close things manually, and didn't try to "declare your variables at the top" C-style:

with BytesIO() as zip_file:
    with zipfile.ZipFile(zip_file, 'a', zipfile.ZIP_DEFLATED, False) as zip:
        zip.writestr('redirects.csv', f_redirects.getvalue())
        zip.writestr('links.csv', f_bad_links.getvalue())
        zip.writestr('metadata.csv', f_metadata.getvalue())
    zip_contents = zip_file.getvalue()

k = Key(BUCKET)
k.key = '%s.zip' % base_name
k.set_metadata('Content-Type', 'application/zip')
k.set_contents_from_string(zip_contents)

If you're using Python 2.x, and want to stay with StringIO, it isn't directly usable as a context manager, so you'll have to replace the first line with:

with contextlib.closing(StringIO()) as zip_file: