Search code examples
pythonnumpyiteratorgeneratorcycle

How does cycle prevent a generator from throwing a StopIteration exception?

Considering this example:

from itertools import cycle

def foo():
    for i in range(3):
        yield i

c = cycle(foo())

next(c)  # -> 0
next(c)  # -> 1
next(c)  # -> 2 [StopIteration should be thrown here normally]
next(c)  # -> 0
...

How does cycle prevent the generator from exciting via a StopIteration? I thought a generator could only be executed once, since the generator only returns its current value and moves on.

Is cycle simply re-creating the generator when StopIteration is thrown? Could this become a problem (inefficiency) when I iterate over for example large numpy arrays?

Secondary question: is this a "pythonic" way of cycling large data sets with iterators/generators? Or should I transfer the cycling logic directly to the generator itself (like defining indexes and using a while loop with resetting indexes)?

My intention is to cycle through large data sets (mostly numpy array; >100.000 entries) efficiently.

Solution

  • How does cycle prevent the generator from exciting via a StopIteration?

    It doesn't. The generator reaches its end and exits with a StopIteration as normal. cycle stores the generator's output, and when cycle sees the StopIteration, it switches to producing items from its stored history of what the generator produced.