Astroscrappy not working on multiprocessing Jupyter

Question

I have some multiprocessing code here that tries to run multiple astroscrappy processes at once. However, everything stops when it actually has to call astroscrappy. I am running this in a jupyter notebook.

def a_test(i, q):
    import astroscrappy
    print(1)
    path = i
    s = fits.getdata(path)
    print(2)

    print(2.5)
    a = astroscrappy.detect_cosmics(s)
    print(3)
    q.put([a, i])
bundle = []
import multiprocessing as mp
queue = mp.Manager().Queue()

processes = [] 
for k, item in enumerate(paths):
    processes.append(mp.Process(target=a_test, args=(item, queue)))
    
# Run processes
for p in processes:
    p.start()
for p in processes:
    bundle.append(queue.get())

It will only print out 1, 2, 2.5, but not 3 which comes after calling astroscrappy. Any ideas why it won't work?

Answer 1

If you look at astroscrappy's github here: https://github.com/astropy/astroscrappy, you will see that it already uses its own multiprocessing called OpenMP. Trying to further multiprocess this kind of function will yield marginal results or not even anything at all. Your best bet is to just stick with the speed you already have, which should be fairly quick.