shutil copy command writes 0 byte file copying temp files

I have another method calling this function 3 times to write 3 different files. The first 2 files are copied to the destination folder as expected, the third file is always zero bytes. If I turn off deletion I can see that all 3 temp files are written to successfully. The code reports no errors, but at the end of the process the third file is always empty.

Does anyone know why this is failing 1 out of 3 times?

def write_file(file_destination: str, content: bytes):
    with tempfile.NamedTemporaryFile() as fp:
        fp.write(content)
        shutil.copy(fp.name, file_destination)

The following variation works, however, I would like to understand why the first two files work and the third one does not in the above code.

def write_file(file_destination: str, content: bytes):
    with tempfile.NamedTemporaryFile(delete=False) as fp:
        fp.write(content)
    shutil.copy(fp.name, file_destination)
    os.remove(fp.name)

Solution

This is because the content has not been written to disk yet, when you perform the copy. This happens because writes are buffered, and do not always happen immediately after you call file.write. To make sure the content is written to disk at a given point, you can use file.flush.

In your case, it is sufficient to change your code to:

def write_file(file_destination: str, content: bytes):
    with tempfile.NamedTemporaryFile() as fp:
        fp.write(content)
        fp.flush()
        shutil.copy(fp.name, file_destination)

For more information on when the contents are actually written to disk you can see the documentation of io.BufferedWriter. The relevant part is:

The buffer will be written out to the underlying RawIOBase object under various conditions, including:

when the buffer gets too small for all pending data;

when flush() is called;

when a seek() is requested (for BufferedRandom objects);

when the BufferedWriter object is closed or destroyed.

Therefore, in your first example, it is possible that it works only sometimes, because those times the content you are writing exceeds the buffer and so needs to be written out immediately.

Your second example, instead, works, because when you are exiting the with block, the file is closed and thus the buffer needs to be flushed out and written to disk.