for loop generator with a sentinel

Question

I have an application with one producer and many consumers, and a queue, which communicates them. The consumer should collect some data from queue, let's say qsize()/number_of_consumers, but it must stop it work when a sentinel appears.

I have such a code:

frame = 0
elems_max = 10
while frame is not None:
    frames = []
    for _ in range(elems_max):
        frame = queue_in.get()
        if frame:
            frames.append(frame)
        else:
            break
   process_data(frames)

As You can see None is a sentinel for this queue, and when it appears I wan't to break my working process. I also want to get more then one element for data processing.

What is the fastest method to achieve this [in python 3.5]?

Answer 1

I understand that you want to break your outer while when encountering a None.

You can hold a boolean variable that is True while the while must execute, and False when it should stop.

This would look like this:

frame = 0
elems_max = 10
running = True
while running and frame is not None:
    frames = []
    for _ in range(elems_max):
        frame = queue_in.get()
        if frame is not None:
            frames.append(frame)
        else:
            running = False
            break
   process_data(frames)

The break instruction will break the inner for, but not the outer while. However, having set running to False, the while loop will stop.

---

Based on your comment.

It is not possible to include a break statement in a comprehension, nor an else clause, as you wanted to do:

frames = [f for i in range(elems_max) if queue_in.get() is not None else break]

However, you can build your list, and then remove all the elements after a None:

frames = [queue_in.get() for _ in range(elems_max)]

try:
    noneId = frames.find(None)
    frames = frames[:noneId]
except ValueError:
    pass

This is not very efficient, because potentially many elements will be appended in frames for nothing. I would prefer a manual construction, to avoid this hazard.

---

One more solution, based on a generator. This might not be what you expected, but the syntax is rather simple, so you may like it.

The idea is to wrap the getting of the data inside of a generator, that breaks on a None value:

def queue_data_generator(queue, count):
    for _ in range(count):
        item = queue.get()
        if item is None:
            raise StopIteration
        else:
            yield item

Then, instantiate this generator, and simply iterate over it:

g = queue_data_generator(queue_in, elems_max)
frames = [frame for frame in g]

The frames list will contain all the frames contained in queue_in, until the first None. The usage is rather simple, but you have to setup it by defining the generator. I think it's pretty elegant though.