How can the CPU affinity for OpenMP threads be set on a (Linux) system with cores removed from the scheduler (e.g. using the isolcpus boot parameter)?
On a system where only the first 12 cores are available to the scheduler (isolcpus=12-127), calling the OpenMP program with
GOMP_CPU_AFFINITY=64-127 OMP_NUM_THREADS=64 taskset --cpu-list 64-127 command
all 64 OpenMP threads run on CPU 64.
With
GOMP_CPU_AFFINITY=64-127 OMP_NUM_THREADS=64 command
All 64 OpenMP threads run on CPUs 0-12.
How can I get the 64 OpenMP threads to run on CPUs 64-127?
I could do it programmatically within the software using
#pragma omp parallel for
for(...)
pthread_setaffinity_np(...)
Is there any way this can be done externally when starting the OpenMP program?
(Note: A similar issue was brought up in Core isolation limits OpenMP to a single core but no solution was provided. The given answer "I think you are asking OpenMP to do something that is impossible." is not true, as all that the OpenMP library needs to do is to call pthread_setaffinity_np() or equivalent functions. It shouldn't even need taskset to run on a CPU specified in GOMP_CPU_AFFINITY.)
It appears that both Intel libiomp and GNU libgomp must be intersecting the affinity mask inherited from the calling process (non-isolated cores 0-11) with any GOMP_CPU_AFFINITY, KMP_AFFINITY or OMP_PLACES specifiers. Hence taskset is needed.
Here is what works for Intel:
KMP_AFFINITY=explicit,proclist=[12-191] taskset --cpu-list 12-191 command
OMP_PLACES="12:180" taskset --cpu-list 12-191 command
OMP_PLACES="{12}:180" taskset --cpu-list 12-191 command
This works for GNU:
GOMP_CPU_AFFINITY=12-191 taskset --cpu-list 12-191 taskset --cpu-list 12-191 command # note the double taskset
OMP_PLACES="{12},{13},(explicitly list all cores),{191}" taskset --cpu-list 12-191 command
OMP_PLACES="{12}:180" taskset --cpu-list 12-191 command
This does not work for Intel:
taskset --cpu-list 12-191 command
KMP_AFFINITY=explicit,proclist=[12-191] command
OMP_PLACES="12:180" command
OMP_PROC_BIND=true OMP_PLACES="12:180" command
This does not work for GNU:
taskset --cpu-list 12-191 command
OMP_PLACES="12:180" taskset --cpu-list 12-191 command
OMP_PLACES="12:180" taskset --cpu-list 12-191 taskset --cpu-list 12-191 command
GOMP_CPU_AFFINITY=12-191 taskset --cpu-list 12-191 command
OMP_PLACES="{12},{13},(explicitly list all cores),{191}" command
OMP_NUM_THREADS=180 and isolcpus=12-191 in all examples.