how to use asyncio with boost.python?

Question

Is it possible to use Python3 asyncio package with Boost.Python library?

I have CPython C++ extension that builds with Boost.Python. And functions that are written in C++ can work really long time. I want to use asyncio to call these functions but res = await cpp_function() code doesn't work.

• What happens when cpp_function is called inside coroutine?
• How not get blocked by calling C++ function that works very long time?

NOTE: C++ doesn't do some I/O operations, just calculations.

Answer 1

What happens when cpp_function is called inside coroutine?

If you call long-running Python/C function inside any of your coroutines, it freezes your event loop (freezes all coroutines everywhere).

You should avoid this situation.

How not get blocked by calling C++ function that works very long time

You should use run_in_executor to run you function in separate thread or process. run_in_executor returns coroutine that you can await.

You'll probably need ProcessPoolExecutor because of GIL (I'm not sure if ThreadPoolExecutor is option in your situation, but I advice you to check it).

Here's example of awaiting long-running code:

import asyncio
from concurrent.futures import ProcessPoolExecutor
import time


def blocking_function():
    # Function with long-running C/Python code.
    time.sleep(3)
    return True


async def main():        
    # Await of executing in other process,
    # it doesn't block your event loop:
    loop = asyncio.get_event_loop()
    res = await loop.run_in_executor(executor, blocking_function)


if __name__ == '__main__':
    executor = ProcessPoolExecutor(max_workers=1)  # Prepare your executor somewhere.

    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    try:
        loop.run_until_complete(main())
    finally:
        loop.run_until_complete(loop.shutdown_asyncgens())
        loop.close()