Search code examples
python-3.xparallel-processingmpi4pympiexec

mpiexec : why is my python program running on only one CPU?

I am trying to parallelize a python program (program_to_parallelize.py) into 16 subprocesses on my 16 cores machine. I use this code, which is part of a Python script :

import subprocess
subprocess.call("mpiexec -n 16 python program_to_parallelize.py", shell=True)

This runs without any error but when I look at CPUs usage, I see that all subprocesses are running on one single CPU. (Click here to see what I get when typing "top 1" in command line) But I would prefer that the 16 processes each take 100% of one CPU rather than all sharing the first one.

I am working on a 16 cores Ubuntu 16.04.6 LTS.
I use version 3.0.3 of mpi4py
I use version 3.3.2 of mpiexec

Solution

  • I figured it out actually. One solution is to bind each process to a CPU after starting the execution. To do this, you can use this command :

    taskset -pc [CPU number] [process ID]

    for example :

    taskset -pc 2 3039

    You can find more details about how to assign a process to a CPU on this website : https://www.hecticgeek.com/2012/03/assign-process-cpu-ubuntu-linux/