we have a machine with 2x64-core CPU, each core consists of 2 virtual cores, so in htop we see 256 distinct (virtual) CPUs. We configured Slurm quality of service to better manage CPU usage per user. I.e. we have defined a --qos=cpus50 which, as far as I understand it, gives me a budget of 50 virtual cores to compute my jobs. I created a test.sbatch script with an array of 100 jobs. Each job takes 10s to compute. So with the following config, I would hope that my jobs will be finished in 20s + some small overhead.
#!/bin/bash
#SBATCH --job-name=test
#SBATCH --qos=cpus50
#SBATCH --array=1-100
#SBATCH --cpus-per-task=1
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --ntasks-per-core=1
#SBATCH --open-mode=append
#SBATCH --output=%x.out
python3 -c "import os; jobid = int(os.getenv('SLURM_ARRAY_TASK_ID')); start = f'JOBID:{jobid:04d} | Start {time.ctime()}'; time.sleep(10); print(f'{start} | End {time.ctime()} |')"
However,running the script above spawns only 25 jobs at once (according to squeue output) and finishes in 47seconds. (2x the desired duration). Running with --ntasks-per-core=2 results in the same behavior. Running with --ntasks=2 and --ntasks-per-core=2 results in the same behavior.
What am I doing wrong? I just want to run 50 jobs at once since I already have the virtual cores available. Thank you
Answering my own question. A member of our group found an answer here.
The problem was in Slurm configuration. In short, for our setup, we had to change the relevant part in slurm.conf from
SelectTypeParameters=CR_Core
NodeName=nodename CPUs=256 Sockets=2 CoresPerSocket=64 ThreadsPerCore=2
to
SelectTypeParameters=CR_CPU
NodeName=nodename CPUs=256
Now the sbatch script from the question spawns 50 jobs and finishes in a bit more than 20s as expected.