Enter singularity container running in slurm job

Question

I'm running following job_script.sh:

#!/bin/bash

singularity --debug run --app ${APP} \
                             --bind "vol/on/host":"vol/on/container" \
                             ${IMG_DIR}/${IMG_NAME}

with sbatch command (it is long running job):

sbatch -p ${QUEUE} --qos=${QOS} --job-name ${JOB_NAME} -o ${JOB_NAME}.o%J -e ${JOB_NAME}.e%J job_script.sh

What I want to achieve is to execute command in running container which is run in following slurm job:

$ squeue -u $USER
JOBID     ARRAY_TASKPARTITION NAME                                  USER      STATE     NODELIST(REASON) CPUS  NODES TRES_PERSTART_TIME            TIME
7988346   N/A       defq      rstudio                               myuser  RUNNING   nc023            1     1     N/A     2019-11-25T06:23:32   2:38:03

Answer 1

When using singularity exec or singularity run, you're spawning a single process and you are unfortunately unable to run additional processes from within the same context as you might with docker run ....

However, you can do this if you use singularity instance start some_instance_name instead. Afterwards, you can use singularity shell or singularity exec to access the same environment as the running job. More details are available in the documentation.