I need to submit a slurm job that needs to have a core count divisible by 7 on a cluster with 64 core nodes. One solution is to run a 7 node/16 core job, which works well because the parallelization works extremely well between these 7 groups of cores (very little communication between the 7 groups).
Scheduling of this job becomes difficult however since its hard for 7 nodes to open up 16 cores at one time. Are there any ways to submit jobs in the following configurations?
Explicitly request 2 nodes, one uses 64 cores and one uses 48 cores.
Allow the job to combine the 7 node job to place multiple node allocations on a single node, allowing it to simply find 7 groups of 16 cores.
The only thing I cannot allow is the groups of 16 cores to be split over 2 nodes, as this will dramatically hurt performance.
This is running on slurm 20.11.8
Explicitly request 2 nodes, one uses 64 cores and one uses 48 cores.
If I understood your requirement correctly, then this will satisfy your first configuration requirement:
srun -N 2 -n 112 -m block:block --cpu-bind=core app_binary