I'm trying to run this multiprocessing pool and I can't figure out why it won't run. It just seems to be processing endlessly. I am confident the function I am calling works (I have tested without the pool) so the error seems to be here. Any thoughts? The code runs as far as the for loop based on what prints.
The function it is calling runs rasterstats.zonal_stats if that matters.
if __name__ == "__main__":
#create and configure the process pool
print('inside', flush=True)
with multiprocessing.Pool(processes = multiprocessing.cpu_count()) as pool:
print('inside2', flush=True)
#prepare arguments for function as list of variables and bands
items = [(var, band) for var in clim_rasts.keys() for band in bands]
print('inside3', flush=True)
#concat the results
stime2 = time.time()
for result in pool.starmap(main_climate_task, items):
print('result', result, flush=True)
climate_data = pd.concat([climate_data, result]) )
etime2 = time.time()
dur2 = etime2-stime2
print(dur2, flush=True)
All due thanks to @Eureka for pointing out the environment may be the issue. I found out here (Python Multiprocessing within Jupyter Notebook) that multiprocessing will not run in JupyterLab on Windows. I saved my code in a .py file and ran that and it works fine.
%%writefile multiprocess.py
(ALL THE CODE GOES HERE)
%run multiprocess.py