In my application, I need to get files from Minio storage and create a Zip-file from them. Some files might be really large so I'm trying to write them in chunks to be able handle the process more efficiently. The result however is a zip file with multiple entries with the same file name. I assume these are the chunks. How can I combine the chunks so that I would only have the original file in the Zip-file? Or is there some better way to handle writing large files into Zip?
This is the code block where I write the chunks:
zip_buffer = io.BytesIO()
with zipfile.ZipFile(zip_buffer, "w") as zip_file:
for url in minio_urls:
file_name = url.split("/")[-1]
# Retrieve the Minio object
minio_object, object_name = get_object_from_minio(url)
stream = minio_object.stream()
while True:
chunk = next(stream, None) # Read the next chunk
if chunk is None:
break
zip_file.writestr(file_name, chunk)
zip_file.writestr() would only be used when you have the entire entry contents to be written at once. If you want to write a chunk of the entry at a time, you need to use ent = zip_file.open(), and then ent.write().