I've written this function that works OK when called in its own jupyter notebook but when I save the notebook as a .py file and call it from another notebook, I get the following error:
UnboundLocalError: local variable 'results' referenced before assignment
def get_results(inputs):
if __name__ == '__main__':
with multiprocessing.Pool() as pool:
results = pool.starmap(aaa.dostuff, inputs)
pool.close()
pool.join()
return results
I thought that pool.close() followed by pool.join() takes care of the order in which the commands are executed, not sure what to do.
You need to remove the if __name__ == '__main__': test from your function. You may have misunderstood what its purpose is when using the multiprocessing module.
Instead, put this at the module level, and put the call to get_results() in the block protected by it:
def get_results(inputs):
with multiprocessing.Pool() as pool:
results = pool.starmap(aaa.dostuff, inputs)
pool.close()
pool.join()
return results
if __name__ == '__main__':
# ... other code you have to produce 'inputs', that needs to run *before*
# starting the pool of child processes, goes here ...
results = get_results(inputs)
# do something with the results.
The if __name__ test is rarely used anywhere but at the top level of a module. The global variable __name__ is set by Python when it imports a module. Only when you run your file as a script (with python path/to/file.py), does Python set that variable to the string "__main__", and it is a handy value to test for when you want to run code only when the current module is the starting point.
Depending on your OS and configuration, when using multiprocessing Python creates new, separate child processes that either start off with a copy of the Python interpreter state, or with a dedicated multiprocessing script specifically designed to run multiprocessing worker code. Your own script is then no longer the main starting point and the value of __name__ will be different.
What went wrong in your case is that you almost certainly called get_results() in a worker process, where __name__ is set to something else. At that point your function executes two lines:
if __name__ == '__main__': - Outcome False, so the block is skippedreturn result - error! Nothing set a value for this local name, so it is still unbound.