Slurm do not limit number of available GPUs in job when --gres is not set

I have a all-in-one node with 4 GPUs. I meet the problem that all GPUs are available for job if the --gres=gpu:n is not set.

The env "$CUDA_VISIBLE_DEVICES" is null. This is expected.

This is my srun command:

srun --mpi=none -n1 -p debug python test_gpu.py

if torch.cuda.is_available():
    # Get the number of available GPUs
    num_gpus = torch.cuda.device_count()

the num_gpus is 4. and all GPUs are available.

This is my slurm.conf:

SlurmctldHost=gpu-server35
GresTypes=gpu
MpiDefault=none
ProctrackType=proctrack/cgroup
ReturnToService=1
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/spool/slurmd
#SlurmUser=slurm
SlurmdUser=root
#SrunEpilog=
#SrunProlog=
StateSaveLocation=/var/spool
SwitchType=switch/none
#TaskEpilog=
TaskPlugin=task/cgroup,task/affinity

#
# SCHEDULING
#DefMemPerCPU=0
#MaxMemPerCPU=0
#SchedulerTimeSlice=30
SchedulerType=sched/backfill
SelectType=select/cons_tres
SelectTypeParameters=CR_Core
#

#
# LOGGING AND ACCOUNTING
AccountingStorageType=accounting_storage/none
#AccountingStorageUser=
AccountingStoreJobComment=YES
ClusterName=cluster
JobCompType=jobcomp/none
#JobCompUser=
#JobContainerType=job_container/none
JobAcctGatherFrequency=30
#JobAcctGatherType=jobacct_gather/none
JobAcctGatherType=jobacct_gather/cgroup #optional for gathering metrics
PrologFlags=Contain                     #X11 flag is also suggested
SlurmctldDebug=info
SlurmctldLogFile=/var/log/slurmctld.log
SlurmdDebug=info
SlurmdLogFile=/var/log/slurmd.log

# COMPUTE NODES
NodeName=gpu-server35 Gres=gpu:tesla:2 CPUs=48 RealMemory=200000 Sockets=2 CoresPerSocket=12 ThreadsPerCore=2 State=UNKNOWN
PartitionName=debug Nodes=gpu-server35 Default=YES MaxTime=INFINITE State=UP

cgroup.conf

CgroupAutomount=yes
CgroupMountpoint=/sys/fs/cgroup
ConstrainCores=yes
ConstrainDevices=yes
ConstrainKmemSpace=no        #avoid known Kernel issues
ConstrainRAMSpace=yes
ConstrainSwapSpace=yes
TaskAffinity=no              #use task/affinity plugin instead

Slurm limit number of available GPUs in job

Solution

The first thing is to make sure the gres.conf file is complete, with the File= parameter properly specified.

Then, if the cgroup configuration is correct, only the GPUs allocated by Slurm should be visible to the job processes. Running nvidia-smi in a GPU-less job on a HPU node should return

No devices were found

If that is no the case, you should double check the cgroup configuration (which appears correct in your case) and make sure all daemons have loaded that configuration (restart them all to be sure).

You can also set export CUDA_VISIBLE_DEVICES= in the Slurm task prolog but beware that this can be overridden by the job program.