Search code examples
pythonpython-3.xparallel-processingtaskpython-asyncio

How to use asyncio for parallel tasks


I was wondering how I could use asyncio to handle tasks similar to what nodeJS does. I want to run tasks at the same time without opening threads. Example:

import asyncio
 
@asyncio.coroutine
def my_coroutine(task_name, seconds_to_sleep=3):
    print('{0} sleeping for: {1} seconds'.format(task_name, seconds_to_sleep))
    yield from asyncio.sleep(seconds_to_sleep)
    print('{0} is finished'.format(task_name))
 
 
loop = asyncio.get_event_loop()
tasks = [
    my_coroutine('task1', 4),
    my_coroutine('task2', 3),
    my_coroutine('task3', 2)]
loop.run_until_complete(asyncio.wait(tasks))
loop.close()

will output:

task1 sleeping for: 4 seconds
task2 sleeping for: 3 seconds
task3 sleeping for: 2 seconds
task3 is finished
task2 is finished
task1 is finished

but when I try to do it with a different task it won't work like that.

import asyncio
import timeit
 
@asyncio.coroutine
def my_coroutine(task_name):
    print('order placed for ' + task_name)
    print(timeit.timeit('1 + 3 ', number=50000000))
    print(task_name + ' done')
 
 
loop = asyncio.get_event_loop()
tasks = [
    my_coroutine('task1'),
    my_coroutine('task2'),
    my_coroutine('task3')]
loop.run_until_complete(asyncio.wait(tasks))
loop.close()

outputs

order placed for task2
0.6677237730912453
task2 done
order placed for task1
0.6627442526498016
task1 done
order placed for task3
0.665618849882418
task3 done

Solution

  • asyncio doesn't run things in parallel. It runs one task until it awaits, then moves on to the next. The sleeps in your first example are what make the tasks yield control to each other. Your second example doesn't await anything, so each task runs until completion before the event loop can give control to another task.

    If you add something awaitable (e.g., asyncio.sleep) into your coroutine, each one will yield control and give the others a chance to run.

    @asyncio.coroutine
    def my_coroutine(task_name):
        print('order placed for ' + task_name)
        yield from asyncio.sleep(0)  # Another coroutine will resume here.
        print(timeit.timeit('1 + 3 ', number=50000000))
        yield from asyncio.sleep(0)  # Another coroutine will resume here.
        print(task_name + ' done')