Search code examples
pythonpython-3.xiobuffer

buffering argument of open() seemingly ignored when working with text I/O


The buffering argument of the built-in open() function seems to be ignored when working with text I/O:

Text I/O:

>>> f = open("myfile_text.txt", "w",buffering=2)
>>> f.write('1')
1
>>> f.write('1')
1
>>> f.write('1')
1

mymachine:~ myuser$ cat myfile.txt ; echo
mymachine:~ myuser$

Binary I/O:

>>> f = open("myfile_binary.txt", "wb",buffering=2)
>>> f.write('1'.encode())
1
>>> f.write('1'.encode())
1
>>> f.write('1'.encode())
1

mymachine:~ myuser$ cat myfile.txt ; echo
11
mymachine:~ myuser$

Why? Is this intended behavior?


Solution

  • I've poked around the a bit. Indeed the documentation states:

    buffering is an optional integer used to set the buffering policy. Pass 0 to switch buffering off (only allowed in binary mode), 1 to select line buffering (only usable in text mode), and an integer > 1 to indicate the size in bytes of a fixed-size chunk buffer...

    Hence it would be reasonable to expect that specifying buffering=2 would cause buffer of size 2B to be used (and see files flushed when using that buffer up). However in the text mode it seem to (and the documentation doesn't really clearly indicate that in this place) affect the underlining BufferedWriter, but TextIOWrapper whose write() you end up calling does still "its own thing"... Starting with Python 3.7 you tell it not to and have it pass the writes immediately to the underlying object by calling f.reconfigure(write_through=True) on the file you have opened.


    Actually all the code and checks for value of write_through are already in Python 3.6 (I've checked 3.6.8) (as a matter of fact _io_TextIOWrapper_write_impl, i.e. _io.TextIOWrapper.write, are identical in both versions), but the write_through attribute is not exposed (yet).