Search code examples
pythonfilesizefilesize

How do I write to file in blocks in order to circumvent an error with large file sizes?


I have this very simple piece of code:

f = open('file.txt','w+')
f.write(result)
f.close()

The issue is that the string 'result' can be up to several gigabytes in size. Although it is successfully created, writing it to a file results in this error:

OSError: [Errno 22] Invalid argument

I've read that this is a known issue with running Python in OS X (I'm running 10.13 High Sierra). How can I break the write function up into blocks to get around it?

(I also know this is an insurmountable problem in 32-bit versions due to inherent limitations, but I'm running 64-bit)


Solution

  • Try writing in chunks

    def chunks(l, n):
        """Yield successive n-sized chunks from l."""
        for i in range(0, len(l), n):
            yield l[i:i + n]
    
    
    with open('file.txt','w+') as f:  
        for chunk in chunks(results, 1024):  # try to play with this number
            f.write(chunk)