Submit Slurm Jobs using Python

Question

I have foo.sh as below

#!/bin/bash
#SBATCH -A research
#SBATCH -p long
#SBATCH --mem-per-cpu=1024
#SBATCH -N 1
#SBATCH -n 24
#SBATCH -t 2-00:00:00
#SBATCH --mail-type=END
#SBATCH --exclude=node37

module load Gaussian/09revC

export GAUSS_SCRDIR=/scratch/$USER.$SLURM_JOBID
/bin/mkdir -p $GAUSS_SCRDIR

g09 $1

/bin/rm -rf $GAUSS_SCRDIR

And I call foo.sh from another bash file called submit.sh

#!/bin/bash
sbatch foo.sh 00_molecule_name_some_method.com
sbatch foo.sh 01_molecule_name_some_method.com
sbatch foo.sh 02_molecule_name_some_method.com

This submits multiple jobs.

I want to change submit.sh to Python file which will call foo.sh provide it with argument which will go to $1 in foo.sh, Python should not wait for the jobs to get completed.

Also molecule_name and some_method are variables which I will provide in for loop

template Python script submit.py

for molecule_name in molecules:
    for some_method in methods:
        foo.sh molecule_name some_method

I get this error:

/bin/mkdir: cannot create directory ‘/scratch’: Permission denied
PGFIO/stdio: No such file or directory
PGFIO-F-/OPEN/unit=11/error code returned by host stdio - 2.
File name = /scratch/USERNAME./Gau-41212.inp
In source file ml0.f, at line number 181
PGFIO/stdio: No such file or directory
PGFIO-F-/OPEN/unit=11/error code returned by host stdio - 2.
File name = /scratch/USERNAME./Gau-41213.inp
In source file ml0.f, at line number 181

Answer 1

Use Subprocess

import subprocess
for molecule_name in molecules:
    for some_method in methods:
        command1 = subprocess.Popen(['foo.sh', molecule_name, some_method])