Search code examples
linuxslurmhpc

How to pipe input to SLURM's sbatch?


I have a job script called testjob.sh which I submit as

sbatch testjob.sh

Is it possible to pipe some input to the script? If I ran as a simple command, I would do

echo 1 2 3 4 | ./testjob.sh

however, with SLURM, I tried the following and it does not work:

echo 1 2 3 4 | sbatch testjob.sh 

My example script is this:

#!/bin/bash
#SBATCH --export=ALL
#SBATCH --job-name=jobname
#SBATCH -e jobname.err
#SBATCH -o jobname.out
#SBATCH -p RM-shared
#SBATCH -n 1
#SBATCH --time=00:05:00

echo "jobscipt"
while read -r input 
do
        echo "from jobscipt $input"
done

and the outputs:

#this one worked
> echo 1 2 3 4 | ./testjob.sh
jobscipt
from jobscipt 1 2 3 4

#slurm did not take the input
> echo 1 2 3 4 | sbatch testjob.sh
Submitted batch job 3758713
> cat jobname.out
jobscipt

I know it's possible to set arguments to sbatch via setting environmental variables with --export, but how would I pipe an input to it? A workaround to pipe would be a temporary file but that's not particularly elegant.


Solution

  • sbatch runs the job in a different environment ("in the background"), therefore you can't pipe stuff into the scripts.

    You can avoid this in two ways:

    • use srun instead of sbatch, which doesn't disconnect the job from your session and piping works more-or-less normally. Still, this doesn't allow you to "queue" the job as with batches.
    • use intermediate file for saving the input, i.e., modify the batch file to read the input from testjob.in:
    while read -r input
    do
       ...
    done < testjob.in
    

    and put your input into the intermediate file before enqueueing the batch:

    echo 1 2 3 4 > testjob.in
    sbatch testjob.sh