Search code examples
pythonyield

yield/return from python generators

Bumped into a behaviour I'm struggling to grasp without an assistance. Here's a recursive function:

OPERATORS = ['+', '-', '*', '/']

def recursive(value, operands):
    if not operands:
        return value
    for operator in OPERATORS:
        new_value = str(eval(value + operator + operands[-1]))
        new_operands = operands[:-1]
        yield from recursive(new_value, new_operands)

def countdown(value, operands):
    return next(i for i in recursive(value, operands) if i == '0')

ans = countdown('5', ['1', '2', '3'])
print(ans)

return value raises StopIteration which is not handled by caller so exception is raised before anything returns.

If return value is substituted by yield value; return like this:

def recursive(value, operands):
    if not operands:
        yield value
        return
    for operator in OPERATORS:
        new_value = str(eval(value + operator + operands[-1]))
        new_operands = operands[:-1]
        yield from recursive(new_value, new_operands)

or yield value; raise StopIteration or yield value; raise StopIteration(value) or loop is hidden under else clause then exception is handled by caller the way I expect and function eventually returns '0'. All of these also raise exception: raise StopIteration, both bare and with argument, yield; return value, and bare return.

In short, caller breaks when StopIteration is raised and yield never returned.

Why?

PEP380 states that first StopIteration is handled in a different way then others. PEP479 says:

Currently, StopIteration raised accidentally inside a generator function will be interpreted as the end of the iteration by the loop construct driving the generator.

Apparently, except the first time. Still, the details of underlying implementation and exact reasoning behind it are unclear to me.

Two more additional questions:

  • what's the right way of writing this snippet? return (yield value)?

  • is it a quirk, a feature, or something else?

Edit: fixed mistakes in code

Edit2: As far as I understand the execution flow in the first snippet is following:

  1. countdown creates generator from recursive('5', ['1', '2', '3'])
  2. Which spawns generators all the way down the tree to recursive('11', [])
  3. At this point StopIteration('11') is raised

And here is the tricky part. What happens here? How StopIteration is handled?

Second snippet:

  1. same
  2. same
  3. '11' is yielded upwards until it reaches countdown
  4. Where it gets rejected by if '11' == '0'
  5. Control flow reaches back to yield '11' and raises StopIteration
  6. Repeat until '0'

From what I see now that is pretty much expected behaviour. StopIteration is intrecepted by it's caller and does not propagate upward. Caller in turn raises StopIteration without arguments. That is why the '11' part of an original exception never reached countdown. Exception in the first snippet traceback was bare StopIteration raised in countdown by recursive('5', ['1', '2', '3'].

Solution

  • StopIteration is raised when you finally run out of operands. Until then, you continue to recur on your list, evaluating results and shortening the list. I think that the yield has returned, but it returned to its caller, which was the previous invocation of recursive, rather than countdown.

    In the second example, you yield a value, and the ensuing call to recursive is what raises StopIteration, as it immediately hits a return.

    As for the example return (yield 42), yes, this is a quirk. The poster of that question was stumbling through making a generator, and discovered that code he later thought was wrong, had actually returned something.