Search code examples
pythonpython-3.xperformancegeneratorbytecode

Why is one generator faster than other with `yield` inside?


I have two functions returning generator:

def f1():
    return (i for i in range(1000))

def f2():
    return ((yield i) for i in range(1000))

Apparently, generator returned from f2() is twice as slower than f1():

Python 3.6.5 (default, Apr  1 2018, 05:46:30) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import timeit, dis
>>> timeit.timeit("list(f1())", globals=globals(), number=1000)
0.057948426001530606
>>> timeit.timeit("list(f2())", globals=globals(), number=1000)
0.09769760200288147

I tried to using dis to see what's going on but to no avail:

>>> dis.dis(f1)
  2           0 LOAD_CONST               1 (<code object <genexpr> at 0x7ffff7ec6d20, file "<stdin>", line 2>)
              2 LOAD_CONST               2 ('f1.<locals>.<genexpr>')
              4 MAKE_FUNCTION            0
              6 LOAD_GLOBAL              0 (range)
              8 LOAD_CONST               3 (1000)
             10 CALL_FUNCTION            1
             12 GET_ITER
             14 CALL_FUNCTION            1
             16 RETURN_VALUE
>>> dis.dis(f2)
  2           0 LOAD_CONST               1 (<code object <genexpr> at 0x7ffff67a25d0, file "<stdin>", line 2>)
              2 LOAD_CONST               2 ('f2.<locals>.<genexpr>')
              4 MAKE_FUNCTION            0
              6 LOAD_GLOBAL              0 (range)
              8 LOAD_CONST               3 (1000)
             10 CALL_FUNCTION            1
             12 GET_ITER
             14 CALL_FUNCTION            1
             16 RETURN_VALUE

Apparently, the results from dis are the same.

So why is generator returned from f1() faster than generator from f2()? And what is proper way to debug this? Apparently dis in this case fails.

EDIT 1:

Using next() instead of list() in timeit reverses the results (or they are the same in some cases):

>>> timeit.timeit("next(f1())", globals=globals(), number=10**6)
1.0030477920008707
>>> timeit.timeit("next(f2())", globals=globals(), number=10**6)
0.9416838550023385

EDIT 2:

Apparently it's bug in Python, fixed in 3.8. See yield in list comprehensions and generator expressions

Generator with yield inside actually yields two values.


Solution

  • Yield in generator expressions is actually a bug, as discussed in this related question.

    If you want to actually see what's going on with dis, you need to introspect on the code object's co_const[0], so:

    >>> dis.dis(f1.__code__.co_consts[1])
      2           0 LOAD_FAST                0 (.0)
            >>    3 FOR_ITER                11 (to 17)
                  6 STORE_FAST               1 (i)
                  9 LOAD_FAST                1 (i)
                 12 YIELD_VALUE
                 13 POP_TOP
                 14 JUMP_ABSOLUTE            3
            >>   17 LOAD_CONST               0 (None)
                 20 RETURN_VALUE
    >>> dis.dis(f2.__code__.co_consts[1])
      2           0 LOAD_FAST                0 (.0)
            >>    3 FOR_ITER                12 (to 18)
                  6 STORE_FAST               1 (i)
                  9 LOAD_FAST                1 (i)
                 12 YIELD_VALUE
                 13 YIELD_VALUE
                 14 POP_TOP
                 15 JUMP_ABSOLUTE            3
            >>   18 LOAD_CONST               0 (None)
                 21 RETURN_VALUE
    

    So, it yields twice.