Search code examples
pythoncpython-c-apicpythonpyobject

Practical use of PyVarObject, the variable-length subtype of cpython’s PyObject


As someone who has been putting the Python C-API through its paces of late, I am curious about PyVarObject, which the documentation says is a subtype of PyObject used specifically for the definition of variable-length types – and though the caveat that:

This type does not often appear in the Python/C API

… there are an abundance of documented support structures, slot hooks, and other API minutiae that are dedicated to PyVarObject, to the point where I am genuinely curious if there are any Python-API-related problems (however specifically niche) for which it can notably ease the solution.


Solution

  • It's mostly for immutable container types (e.g. tuple and bytes). The idea is that the type can have an array at the end of it that contains all its data. When it is allocated (with PyObject_NewVar or similar) extra space is allocated for the (immutable) data it will contain, which simplifies memory management and presumably gains a bit of speed by only having to do a single memory allocation. Because the size never changes after construction it can take this shortcut without having to worry about invalidating references to it by reallocating the memory.

    To confuse things slightly list is also a PyVarObject despite being mutable. However, it has an item size of 0 and so no extra space is actually allocated at the end of the object. Instead it stores a pointer to an array of PyObject*, allocated separately (which can be reallocated if the list grows without invalidating references to the list). Presumably it was made a PyVarObject just to have an ob_size field.