Search code examples
bashparallel-processingcluster-computingsungridengine

Sun Grid Engine - Script launching an Array-jobs and waiting for all of them to finish


When you submit an arrayjob, you do something like this:

qsub -t 1-1000 -N myArray arrayjob.sh 
#This execute arrayjob 1000 simultaneus times, calling the job "myArray"

I would like to wrap this in another script that tells me "Ehi, all the tasks are done!"

Some people will propose:

qsub somethingAfterAllTheTask.sh -hold_jid "myArray"

But

  • I don't want to start another job to execute after myArray: I want to resume the script from which I executed the qsub of myArray
  • myArray is gonna die after launching all the tasks, not after all those tasks are done: I have to wait for the tasks' completion, not for myArray's

So what I want is a pseudo-script like this:

#!/bin/bash
qsub -t 1-1000 -N myArray arrayjob.sh
if [all_the_task_are_completed]; then
    echo Done!
fi

How do I obtain such behavior? Thanks in advance

There is this question but it's not about array-jobs, and it still does not satisfy my problem


Solution

  • Using -sync y when submitting the arrayjob, the command which launches all the tasks will not die until all the tasks are finished.

    Thanks to @Oo.oO for the tip