Set max concurrency with asyncio.create_subprocess_exec

Question

I need to run a program about 500 times with different inputs. I'd like to use asyncio.create_subprocess_exec and want to limit the number of processes running at the same time so as not to clog up the machine. Is there a way to set the concurrency level? For example, I'd expect something like AbstractEventLoop.set_max_tasks.

Answer 1

In situations where one wants to schedule enough program calls that creating each as a coroutine right away starts being an issue, a semaphore-backed task queue as in this answer can be useful:

tasks = TaskQueue(NUM_PARALLEL_PROCESSES)
for input in MANY_INPUTS:
     await tasks.put(run_program(input))

tasks.join()