Search code examples
pythonpython-2.7memoryiosize

Where does StringIO allocates memory


Probably silly question: Where does StringIO allocates memory?

from cStringIO import StringIO
import sys


buff = StringIO()
buff.write('This goes into the buffer. Don"t know what to say more.')
print(buff.__sizeof__())
buff.write('This goes next. blablablabla!!1!!!!')
print(sys.getsizeof(buff))

>> 56
>> 56

I know about .tell() method. But I wonder about how object represents in memory.


Solution

  • If you want to see how it's working under the hood, you can read the source for io.StringIO in the CPython codebase: https://github.com/python/cpython/blob/main/Modules/_io/stringio.c

    I found the answer to my question, which is "Does StringIO(initial_value) copy the underlying string to the buffer (if you only read it)", which is YES (look at _io_StringIO___init___impl(), from line 732 (https://github.com/python/cpython/blob/main/Modules/_io/stringio.c#L732), and then at write_str() https://github.com/python/cpython/blob/main/Modules/_io/stringio.c#L177)

    Going back to your question, there is the C-struct stringio, which has the value buf_size -- this is the real number of allocated bytes in the StringIO, which may be greater than the number of bytes you've put in!

    Actually, StringIO overallocates just in case, about 1.125 times, foreseeing future additions to the buffer, https://github.com/python/cpython/blob/main/Modules/_io/stringio.c#L99

    Unfortunatelly, we don't have access to the buf_size struct member from Python. If you want to keep track of how much you've written, either sum the returns from .write(), believe what .tell() tells you, or get the string out and check it's length:

    len(buff.getvalue())