Python 3.7 - Fast way to concatenate strings and write them to disk

Question

Is there a better implementation for:

file = open("./myFile.txt","w+")
for doc in allDocuments: #around 35 million documents
    file.write(str(doc.number)+"\r\n")
f.close()

Currently this implementation is taking 20sec per file

Answer 1

You could buffer the output and write in blocks:

with open("./myFile.txt", "w") as output:
    lines = []
    for doc in allDocuments:
        lines.append(f"{doc.number}\r\n")
        if len(lines) > 1000:
            output.writelines(lines)
            lines = []
    output.writelines(lines)