I'm running several processes with mpirun in a node with 36 cpus and 72 threads (hyperthreading). When I run them with different mpirun calls (each with -np 2) they run in the same cpus, decreasing the efficiency of the processes. With ps command I see that when I send 7
mpirun -np 2 ./foo
calls, all the processes are running in just 4 threads. Anyone knows how to solve it? I want to distribute the processes across all threads for maximize the efficiency.
The default with Open MPI is to bind MPI tasks to cores with invoked with -np 2. There is no connection between jobs, so all MPI jobs get pinned to the same two cores (0 and 1 iirc).
A suboptimal solution is to avoid binding, and let the Linux scheduler takes care of that
mpirun --bind-to none -np 2 ./foo
A better solution is to use a resource manager such as SLURM, configure cpuset or similar so all your jobs get allocated on disjoint set of two cores.