I have a job script called testjob.sh
which I submit as
sbatch testjob.sh
Is it possible to pipe some input to the script? If I ran as a simple command, I would do
echo 1 2 3 4 | ./testjob.sh
however, with SLURM, I tried the following and it does not work:
echo 1 2 3 4 | sbatch testjob.sh
My example script is this:
#!/bin/bash
#SBATCH --export=ALL
#SBATCH --job-name=jobname
#SBATCH -e jobname.err
#SBATCH -o jobname.out
#SBATCH -p RM-shared
#SBATCH -n 1
#SBATCH --time=00:05:00
echo "jobscipt"
while read -r input
do
echo "from jobscipt $input"
done
and the outputs:
#this one worked
> echo 1 2 3 4 | ./testjob.sh
jobscipt
from jobscipt 1 2 3 4
#slurm did not take the input
> echo 1 2 3 4 | sbatch testjob.sh
Submitted batch job 3758713
> cat jobname.out
jobscipt
I know it's possible to set arguments to sbatch
via setting environmental variables with --export
, but how would I pipe an input to it? A workaround to pipe would be a temporary file but that's not particularly elegant.
sbatch
runs the job in a different environment ("in the background"), therefore you can't pipe stuff into the scripts.
You can avoid this in two ways:
srun
instead of sbatch
, which doesn't disconnect the job from your session and piping works more-or-less normally. Still, this doesn't allow you to "queue" the job as with batches.testjob.in
:while read -r input
do
...
done < testjob.in
and put your input into the intermediate file before enqueueing the batch:
echo 1 2 3 4 > testjob.in
sbatch testjob.sh