Search code examples
pycharmspyderpickleconcurrent.futures

Example from concurrent.futures documentation gives Pickling error in PyCharm, but not Spyder when same interpretor/environment used


I've been fighting to parallelise some code for a while due to Pickling errors. I'm aware of proposed work arounds using dill or cloudpickle (e.g. here or here or here), but I didn't get any of these to work. In my frustration, I have gone to the concurrent.futures official document page and tried running the minimum working example they give for a ProcessPoolExecutor (see here). Specifically:

import concurrent.futures
import math

PRIMES = [
    112272535095293,
    112582705942171,
    112272535095293,
    115280095190773,
    115797848077099,
    1099726899285419]

def is_prime(n):
    if n < 2:
        return False
    if n == 2:
        return True
    if n % 2 == 0:
        return False

    sqrt_n = int(math.floor(math.sqrt(n)))
    for i in range(3, sqrt_n + 1, 2):
        if n % i == 0:
            return False
    return True

def main():
    with concurrent.futures.ProcessPoolExecutor() as executor:
        for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
            print('%d is prime: %s' % (number, prime))

if __name__ == '__main__':
    main()

Running this gives the now all to familiar Pickling error: _pickle.PicklingError: Can't pickle <function is_prime at 0x000001F26AAD6E50>: attribute lookup is_prime on __main__ failed I can't really imagine that the official documentation would be wrong, so what am I doing wrong here? If the documentation is somehow outdated, what is the way to work around this?

I'm using Python 3.9 in PyCharm on Windows if that matters.

I have tried running the code in Spyder and it works without any error. I have changed the run configuration within PyCharm to use exactly the same python interpretator (verified with sys.executable) and environment (as verified with pip list) as Spyder is using, but yet the error is still thrown.


Solution

  • Uncheck the "Run with Python Console" option in the run-configuration used to execute this script (top-menu "Run" -> "Edit Configurations...")

    enter image description here