Python Memory Addresses: Variation Across Python 3.5 - 3.9 on Linux

Question

I stumbled upon a perplexing issue. While trying to manually close out a XDisplay using ctypes.CDLL("libX11.so") to hot-fix VTK, I tried using the Python provided address of the XDisplay. This worked fine on Python 3.8, but caused a segfault on Python 3.7. Here's the code to demonstrate the problem of addressing the XDisplay created within VTK. Specifically:

disp_id = self.ren_win.GetGenericDisplayId()
disp_id_int = int(disp_id[1:].split('_')[0], 16)
print(disp_id, disp_id_int)
x11 = ctypes.CDLL("libX11.so")
x11.XCloseDisplay(disp_id_int)

Which, when run, returns:

user@host:~/tmp$ python3.7 tmp19.py
0000558f843c5930_p_void 94074887231792
Current thread 0x00007f8145824740 (most recent call first):
  File "/home/alex/python/pyvista/pyvista/plotting/plotting.py", line 2769 in close
  File "/home/alex/python/pyvista/pyvista/plotting/plotting.py", line 4207 in show

user@host:~/tmp$ python3.8 tmp19.py
0000000003568de0_p_void 56004064

As you can see, the address from Python 3.8 can be used with libX11.so, but not with Python 3.7 to close the XDisplay window.

As noted in the comments, the memory addresses returned from Python are always relative to the current Python process. However, if this is the case, why do some version of Python appear to return very large addresses for id(True), a structure created upon initialization of Python. Even more perplexing, the versions of Python that return these large values also cause segmentation values when interfacing with the "libX11.so" library.

Here's the results for Python 3.5 - Python 3.9:

user@host:~$ python3.5 -c "print(id(True))"
93962585975264
user@host:~$ python3.6 -c "print(id(True))"
10302848
user@host:~$ python3.7 -c "print(id(True))"
94134164615424
user@host:~$ python3.8 -c "print(id(True))"
11498848
user@host:~$ python3.9 -c "print(id(True))"
11374464

---

Two questions:

• Is this expected behavior? If so, I imagine there's a compiler flag that's been switched on/off for the version that I've installed locally.
• Is there a way to adjust the memory addresses to get a consistent result for all versions of Python?

Answer 1

you're depending on a cpython implementation detail that the id(...) function returns the memory address of the PyObject

in cpython, Py_True and Py_False are macros, pointing at global instances of bool structs:

/* Use these macros */
#define Py_False ((PyObject *) &_Py_FalseStruct)
#define Py_True ((PyObject *) &_Py_TrueStruct)

the actual definition of these structs:

/* The objects representing bool values False and True */

struct _longobject _Py_FalseStruct = {
    PyVarObject_HEAD_INIT(&PyBool_Type, 0)
    { 0 }
};

struct _longobject _Py_TrueStruct = {
    PyVarObject_HEAD_INIT(&PyBool_Type, 1)
    { 1 }
};

the memory address of C globals is ~determined by the compiler, linker, and dynamic loader and depending on a constant value there is undefined behaviour