cv2 reading video frames causes python generator to delay output till the end

Question

I have a python generator that outputs cv2 images (frames) from a video and am processing it using concurrent.futures.ThreadPoolExecutor. I am seeing that tqdm hangs until the generator is exhausted. However, if I feed the executor from gen_frames_dummy, tqdm updates with each task completion.

Any assistance would be greatly appreciated.

def gen_frames(s):
    for frame_num in frames_2_get:
        # skip to frame in video
        s.set(cv2.CAP_PROP_POS_FRAMES, frame_num-1)
        res, frame = s.read()
        if res:
            yield frame
def gen_frames_dummy(s):
    for frame_num in frames_2_get:
        yield np.zeros((16,16,3))
def frame_op(f):
    # process the frame
    pass
with ThreadPoolExecutor() as executor:
    frames_g = gen_frames(vid_capture)
    list(tqdm(executor.map(frame_op, frames_g), total=len(frames2get))

Answer 1

map waits until all workers are completed and then the results are passed to tqdm in your code.

I guess you can rewrite like this to make tqdm work as intended.

    for args in tqdm(frames_g, total=len(frames_2_get)):
        executor.submit(frame_op, args)