How to inspect Generator type object?

Question

With below code (first case),

def f():
   mylist = range(3)
   for i in mylist:
      yield i*i

Without inspecting y, could you say, y=f() returns (x*x for x in range(3)) object of collections.abc.Generator type?

---

With the below code (second case),

def func():
    x = 1
    while 1:
        y = yield x
        x += y

What is that Generator type object returned when invoking y=func()? How do you inspect y to see the code?

Answer 1

First case -- Simple Generator

The generator expression (x*x for x in range(3)) is roughly the same as the simple generator function you described. However, scoping for the genexp can be a little more complicated (which is why we usually recommend that you consume generator expressions immediately rather than passing them around).

Second Case -- Enhanced Generators

The code with the y = yield x is an example of an enhanced generator which is used to send data into a running generator, essentially creating a two-way communication channel between the running generator and the calling code.

Principal use cases for the send/receive logic are to implement coroutines and generator trampolines. See this trampoline example from David Beazley.

Enhanced generators are the key to Twisted Python's beautiful Inline Callbacks which implement coroutines.

How to Examine the Generator

For the variable y in y = func(), the only inspection technique is to examine the public API:

>>> y = func()
>>> dir(y)
['__class__', '__delattr__', '__doc__', '__format__',
 '__getattribute__', '__hash__', '__init__', '__iter__',
 '__name__', '__new__', '__reduce__', '__reduce_ex__',
 '__repr__', '__setattr__', '__sizeof__', '__str__',
 '__subclasshook__', 'close', 'gi_code', 'gi_frame',
 'gi_running', 'next', 'send', 'throw']

How to Inspect the Generator Function

For the generator function itself, you can use the dis module to inspect the code to see how it works:

>>> def func():
        x = 1
        while 1:
            y = yield x
            x += y

>>> import dis
>>> dis.dis(func)
  3           0 LOAD_CONST               1 (1)
              3 STORE_FAST               0 (x)

  4           6 SETUP_LOOP              21 (to 30)

  5     >>    9 LOAD_FAST                0 (x)
             12 YIELD_VALUE         
             13 STORE_FAST               1 (y)

  6          16 LOAD_FAST                0 (x)
             19 LOAD_FAST                1 (y)
             22 INPLACE_ADD         
             23 STORE_FAST               0 (x)
             26 JUMP_ABSOLUTE            9
             29 POP_BLOCK           
        >>   30 LOAD_CONST               0 (None)
             33 RETURN_VALUE 

Tracing the Code with a Debugger

You can use the pdb debugger to trace through the code step-by-step.

>>> import pdb
>>> y = func()
>>> pdb.runcall(next, y)
> /Users/raymond/Documents/tmp.py(2)func()
-> x = 1
(Pdb) s
> /Users/raymond/Documents/tmp.py(3)func()
-> while 1:
(Pdb) s
> /Users/raymond/Documents/tmp.py(4)func()
-> y = yield x
(Pdb) p locals()
{'x': 1}
(Pdb) s
> /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/bdb.py(440)runcall()
-> self.quitting = 1
(Pdb) s
1
>>> pdb.runcall(y.send, 10)
> /Users/raymond/Documents/tmp.py(5)func()->1
-> x += y
(Pdb) s
> /Users/raymond/Documents/tmp.py(4)func()->1
-> y = yield x
(Pdb) locals()
{'__return__': 1, 'x': 11, 'y': 10}