Search code examples
pythonmultithreadingpython-multithreadingconcurrent.futures

pow function blocking all threads with ThreadPoolExecutor


Even though the first example about the ThreadPoolExecutor uses the pow function ( https://docs.python.org/3/library/concurrent.futures.html ) it seems that using pow abnormally blocks all threads when called. See code below.

from concurrent.futures import ThreadPoolExecutor
import time

executor = ThreadPoolExecutor(max_workers=16)

def blocking():
    i = 0
    while True:
        time.sleep(1)
        i+=1
        print("block",i)
        if i>3:
            print("Starting pow....")
            break
    block= pow(363,100000000000000)
    return True

def non_blocking():
    i = 0
    while True:
        time.sleep(1)
        i+=1
        print("non",i)

    return True

f1 = executor.submit(blocking)
f2 = executor.submit(non_blocking)

I expected the output:

block 1
non 1
block 2
non 2
block 3
non 3
block 4
Starting pow....
non 4
non 5
non 6
non 7
non 8

but the program stops running after "starting pow...." giving the result:

block 1
non 1
block 2
non 2
block 3
non 3
block 4
Starting pow....

Solution

  • Your python implementation (probably CPython) must have a Global Interpreter Lock (GIL)

    pow is a native Python function, which doesn't release the GIL when called, efficiently blocking all executions while it's running and it's running quite a while.

    If you want non-blocking, use ProcessPoolExecutor instead. A separate process means 2 separate GILs and not blocking. It has the exact same interface. It also has more constraints as it requires parameters to be picklable (no convenient shared memory like with threads)

    Note that this computation probably never ends or ends up with "out of memory error": since it's power between integers with a big power value, so it doesn't overflow like floats would do but computes and computes, creating bigger and bigger integers each time

    Estimated numbers of digits for this number is roughly:

    >>> math.log10(363)*100000000000000
    255990662503611.25
    

    10^14 digits that is more than any current computer can handle memory wise, and also CPU-wise.