I have an issue with the quantity of available memory in page locked on CentOS 7. After allocating nodes with slurm, when I launch a job with MPI (mvapich), I encounter the following error:
Fatal error in MPI_Init:
Other MPI error, error stack:
MPIR_Init_thread(514).......:
MPID_Init(359)..............: channel initialization failed
MPIDI_CH3_Init(401).........:
MPIDI_CH3I_RDMA_init(221)...:
rdma_setup_startup_ring(410): cannot create cq
It seems to be due to a lack of locked memory. However, it seems to be set to unlimited since ulimit -a returns:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 254957
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4096
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
When launching mpirun with sudo, it is working.
The problem came from slurm which did not retrieve the proper value for max locked memory. salloc -N ulimit -lreturned 64 instead of unlimited.
The solution is to add the follwing line in /etc/init.d/slurm
ulimit -l unlimited
Then stop and start again slurm:
sudo /etc/init.d/slurm stop
sudo /etc/init.d/slurm start