Multiprocessing with Screen and Bash

Question

Running a python script on different nodes at school using SSH. Each node has 8 cores. I use GNU Screen to be able to detach from a single process.

Is it more desirable to:

1. Run several different sessions of screen.
2. Run a single screen process and use & in a bash terminal.

Are they equivalent?

I am not sure if my experiments are poorly coded and taking an inordinate amount of time (very possible) OR my choice to use 1. is slowing the process down considerably. Thank you!

Answer 1

With bash I imagine you're doing something like this (assuming /home is under network mount):

#!/bin/bash    

for i in {1..$NUM_NODES}
do
    ssh node$i 'python /home/ryan/my_script.py' &
done

Launching this script from behind a single screen will work fine. Starting up several sessions of screen provides no performance gains but adds in the extra complication of starting multiple screens.

Keep in mind that there are much better ways to distribute load across a cluster (e.g. if someone else is using up all of node7 you'd want a way to detect that and send your job elsewhere). Most clusters I've worked with have Torque, Maui or the qsub command installed. I suggest giving those a look.