Last night, I sent a large number of jobs with qsub of the same executable but with different input parameters. Most of the jobs were in queue, waiting for the others to finish. This morning, I realized that all the jobs that were in queue used the last instance of my input file.
What is the standard way of working around this issue? Should I have one input file per job and compile my code so it reads the correct one? Or is there a better/more robust solution?
You could create a master PBS script which loops over the different input paramters, executes them either in parallel or sequentially:
this simply gives executable a different input number for each job (IN), you should change this to loop over one or more of your input parameters as needed.
# PBS -l mppwidth=2048
NIN=10 # number of input parameters
for IN in `seq -w 1 $NIN`; do
cd "sub_job_${IN}"
executable $IN # runs jobs sequentially (you might have to prefix this with aprun)
done
or in parallel:
# PBS -l mppwidth=2048
# ^^ these should now be shared among the jobs.
NIN=10 # number of input parameters
for IN in `seq -w 1 $NIN`; do
cd "sub_job_${IN}"
executable $IN & # runs the job in the background, you might
# have to prefix this with `aprun -n .. -N ..` or something
# so that each job only uses a portion of the total
# requested CPUs.
done
wait # wait for all jobs to finish