I have an 8-core RHEL Linux machine running R 4.0.2.
If I ask R for the number of cores, I can confirm that 8 are available.
> print(future::availableWorkers())
[1] "localhost" "localhost" "localhost" "localhost" "localhost" "localhost"
[7] "localhost" "localhost"
> print(parallel::detectCores())
[1] 8
However, if I run this simple example
f <- function(out=0) {
for (i in 1:1e10) out <- out + 1
}
output <- parallel::mclapply(1:8, f, mc.cores = 8)
my top indicates that only 1 core is being used (so that each worker is using 1/8th of that core, or 1/64th of the entire machine).
%Cpu0 :100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 2.0 us, 0.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu4 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu5 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu6 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu7 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 32684632 total, 28211076 free, 2409992 used, 2063564 buff/cache
KiB Swap: 16449532 total, 11475052 free, 4974480 used. 29213180 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3483 user 20 0 493716 57980 948 R 1.8 0.2 0:18.09 R
3479 user 20 0 493716 57980 948 R 1.5 0.2 0:18.09 R
3480 user 20 0 493716 57980 948 R 1.5 0.2 0:18.08 R
3481 user 20 0 493716 57980 948 R 1.5 0.2 0:18.09 R
3482 user 20 0 493716 57980 948 R 1.5 0.2 0:18.09 R
3484 user 20 0 493716 57980 948 R 1.5 0.2 0:18.09 R
3485 user 20 0 493716 57980 948 R 1.5 0.2 0:18.09 R
3486 user 20 0 493716 57980 948 R 1.5 0.2 0:18.09 R
Does anyone know what might be going on here? Another StackOverflow question that documents similar behavior is here. It's clear that I messed up the install somehow. I followed these install instructions for RHEL 7. I'm guessing there is a dependency missing, but I have no idea where to look. If anyone has any ideas of diagnostics to run, etc., they would be most appreciated.
For further context, I have R 3.4.1 also installed on my machine, and when I run this code, everything works fine. (I installed that version through yum.)
I also installed R 4.0.3 yesterday using the same instructions linked above, and it suffers from the same problem.
First run
system(sprintf("taskset -p 0xffffffff %d", Sys.getpid()))
then your simple example
f <- function(out=0) { for (i in 1:1e10) out <- out + 1 }
output <- parallel::mclapply(1:8, f, mc.cores = 8)
works on all 8 cores.