In Python multiprocessing. Pool, how to get the print result in the subprocess

Question

The thing is that:

def main_fun(x):
    ...
    print(x)

if __name__ == "__main__":
    with Pool(5) as pool:
        pool.map(main_fun,range(10000))
    pool.close()
    pool.join()

My question is that: if I run the code on my own computer, it output subprocess print result. But when I submit it as a job to the cluster, I could not see the print result until the whole programs finished. How could I fix it? By the way, the cluster uses the Slurm.

Answer 1

Try doing print(x, flush = True) instead of just print(x).

Flush-variant of call does immediate flush of buffers so that printing is seen right away. While non-flush variant may keep string inside buffer until later time when it is flushed to screen.

Also always do pool.close() and pool.join() inside with block (at the end), not out-side like you did. For the case of just .map() function it doesn't matter, because it is blocking call, but for some other cases it is important to close/join inside with block.