The first time I use next to get the elements in the iterator, an exception is thrown

Question

There are two .py files

test.py

# coding=utf-8
from itertools import chain

def task():
    c = [1, 21, 31, 4, 51, 61, 71]
    d = ['a1', 'b1', 'c', 'd', 'e1', 'f', 'g1']
    e = chain(c, d)
    return id(e)

test1.py

# coding=utf-8
from test import task
import _ctypes

obj_id = task()
tk = _ctypes.PyObj_FromPtr(int(obj_id))

next(tk)

An exception occurred while running the test1 script, like this:

StopIteration

I need to return an object address in a script, and get the object by object address in another script.

Remarks: Is this idea feasible?

Thank you very much！

Answer 1

What you're attempting is Undefined Behavior. Here's a simpler example:

code00.py:

#!/usr/bin/env python3

import sys
import _ctypes


def func():
    custom_object = [1, 2]
    return id(custom_object)


def main():
    addr = func()
    print("Address: {0:d} (0x{1:016X})".format(addr, addr))
    o = _ctypes.PyObj_FromPtr(addr)
    print(type(o))
    print(o)
    print(dir(o))



if __name__ == "__main__":
    print("Python {0:s} {1:d}bit on {2:s}\n".format(" ".join(item.strip() for item in sys.version.split("\n")), 64 if sys.maxsize > 0x100000000 else 32, sys.platform))
    main()
    print("\nDone.")

Output:

[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q057805717]> "e:\Work\Dev\VEnvs\py_064_03.07.03_test0\Scripts\python.exe" code00.py
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] 64bit on win32

Address: 2221013745352 (0x000002051EBC3EC8)
<class 'list'>

[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q057805717]> echo %errorlevel%
-1073741819

[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q057805717]> "e:\Work\Dev\VEnvs\py_064_02.07.15_test0\Scripts\python.exe" code00.py
Python 2.7.15 (v2.7.15:ca079a3ea3, Apr 30 2018, 16:30:26) [MSC v.1500 64 bit (AMD64)] 64bit on win32

Address: 57931144 (0x000000000373F588)
<type 'list'>
[[...]]
['__add__', '__class__', '__contains__', '__delattr__', '__delitem__', '__delslice__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getslice__', '__gt__', '__hash__', '__iadd__', '__imul__', '__init__', '__iter__', '__le__', '__len__', '__lt__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__rmul__', '__setattr__', '__setitem__', '__setslice__', '__sizeof__', '__str__', '__subclasshook__', 'append', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse', 'sort']

Done.

As seen, the program:

• Python 3: crashed
• Python 2: Didn't crash but the "reconstructed" object is garbage

Why?

1. custom_object is created in func
2. Its address is being returned
3. But when the function returns, the object is also scheduled for deletion (which may or may not happen before the next step, depending on gc)
4. The object address (which is a dangling pointer) is used

The most obvious way of getting around this, is to take the object (whose address you want to return) outside of the function, so that it's still valid when its address will be used. Here's your example (I kept it all in one file, but you can split it in 2).

code01.py:

#!/usr/bin/env python3

import sys
import _ctypes
import itertools


c = [1, 21, 31, 4, 51, 61, 71]
d = ["a1", "b1", "c", "d", "e1", "f", "g1"]
custom_object = itertools.chain(c, d)


def func():
    global custom_object
    return id(custom_object)


def main():
    addr = func()
    print("Address: {0:d} (0x{1:016X})".format(addr, addr))
    o = _ctypes.PyObj_FromPtr(addr)
    print(type(o))
    print(o)
    print(dir(o))
    try:
        while True:
            print(next(o))
    except StopIteration:
        pass



if __name__ == "__main__":
    print("Python {0:s} {1:d}bit on {2:s}\n".format(" ".join(item.strip() for item in sys.version.split("\n")), 64 if sys.maxsize > 0x100000000 else 32, sys.platform))
    main()
    print("\nDone.")

Output:

[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q057805717]> "e:\Work\Dev\VEnvs\py_064_03.07.03_test0\Scripts\python.exe" code01.py
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] 64bit on win32

Address: 1519278021096 (0x00000161BC06D9E8)
<class 'itertools.chain'>
<itertools.chain object at 0x000002B3795337F0>
['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__', '__le__', '__lt__', '__ne__', '__new__', '__next__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', 'from_iterable']
1
21
31
4
51
61
71
a1
b1
c
d
e1
f
g1

Done.

Notes:

• This solves your current problem. But anyway the question is kind of unusual. Why do you need it? There should be other (cleaner) ways of achieving your goal. This is a signal of either bad design or an XY Problem. Check [SO]: Is it possible to dereference variable id's?
• You're not relying on [Python 3.Docs]: ctypes - A foreign function library for Python public API, meaning that _ctypes.PyObj_FromPtr might not be available in future Python versions, or its behavior might change without notice
  • Usually ctypes functions work with ctypes objects only. It's a bit strange that this one is working with any kind of object (although it returns a PyObject (which is "the father of everything" in CPython implementation) pointer)