Search code examples
cluster-computingqsubtorque

Array job with unknown task number

I would like to submit an array job on a cluster running SGE. I know how to use array jobs with the -t option (for instance, qsub -t 1-1000 somescript.sh).

What if I don't know how many tasks I have to submit? The idea would be to use something like (not working):

qsub -t 1- somescript.sh

The submission would then go for all the n tasks, with unknown n.

Solution

  • No, open-ended arrays are not a built-in capability (nor can you add jobs to an array after initial submission).

    I'm guessing about why you want to do this, but here's one idea for keeping track of a group of jobs like this: specify a shared name for the set of jobs, appending a counter.

    So, for example, you'd include -N myjob.<counter> in your qsub (or add a #PBS script line for it):

    -N myjob.1 -N myjob.2 ... -N myjob.n