Search code examples
rparallel-processinglapplymclapply

how to track progress in mclapply in R in parallel package


My question is related to this question. However the question referenced above uses multicore package which was replaced by parallel. Most of the functions in the response cannot be replicated in the parallel package. Is there a way to track progress in mclapply. In looking at the mclapply documentation, there is a parameter called mc.silent, I'm not sure if this would be able to track progress, and if so how and where we can see the log file? I'm running on ubuntu linux OS. Please see below for a reproducible example for which I would like to tack progress.

require(parallel) 

wait.then.square <- function(xx){
  # Wait for one second
  Sys.sleep(2);
  # Square the argument 
  xx^2 } 

output <- mclapply( 1:10, wait.then.square, mc.cores=4,mc.silent=FALSE)

Any help would be greatly appreciated.


Solution

  • This is an update of my related answer.

    library(parallel)
    
    finalResult <- local({
      f <- fifo(tempfile(), open="w+b", blocking=T)
      if (inherits(parallel:::mcfork(), "masterProcess")) {
        # Child
        progress <- 0.0
        while (progress < 1 && !isIncomplete(f)) {
          msg <- readBin(f, "double")
          progress <- progress + as.numeric(msg)
          cat(sprintf("Progress: %.2f%%\n", progress * 100))
        } 
        parallel:::mcexit()
      }
      numJobs <- 100
      result <- mclapply(1:numJobs, function(...) {
        # Do something fancy here... For this example, just sleep
        Sys.sleep(0.05)
        # Send progress update
        writeBin(1/numJobs, f)
        # Some arbitrary result
        sample(1000, 1)
      })
      close(f)
      result
    })
    
    cat("Done\n")