How to build a table/tibble/df for parallel execution time for multiple sources?

Question

Suppose I have a set of scripts with the following inside: Each script print into console 'hello world' and script execution time.

t0<- Sys.time()

print('hello world')

t1<- Sys.time()
time.taken<- round(as.numeric(t1-t0), 2)
time.taken
cat(paste("\n[script1] -","Execution time: ", time.taken, "seconds"))

Now I am trying to get the time execution for multiple sources:

source("modules/script1.R")
source("modules/script2.R")
source("modules/script3.R")
source("modules/script4.R")
source("modules/script5.R")
source("modules/script6.R")

However this will print the execution time separately for each script and in the console not in the environment, Would be great to get something like this inside R environment:

#script  execution time
#1             2 seconds
#2             2 seconds
#3             2 seconds
#4             2 seconds
#5             2 seconds
#6             2 seconds

Is this even possible? What resources are available for this kind of processing?

Answer 1

You could use parLapply on Windows or mclapply from parallel package to run the scripts in parallel.

The last value calculated in the script is passed to the source function.

If you make sure the last value of the script is its execution time, it will be returned as a list by lapply:

library(parallel)

l <- 1:6
# Create script files
createscript <- function(numscript) {
  cat('
    t0<- Sys.time()
    print("hello world")
    Sys.sleep(runif(1))
    t1<- Sys.time()
    time.taken<- round(as.numeric(t1-t0), 2)
    time.taken
    print(paste("[script1] -","Execution time: ", time.taken, "seconds")) 
    data.frame(time.taken)'
    ,file =paste0("Script",numscript,".R") , append = F)
  T
}


lapply(l,createscript)

# Scripts list
scripts <- paste0("Script",l,".R")

# Run scripts in parallel
cl <- makeCluster(getOption("cl.cores", detectCores() - 1))
result <- parLapply(cl, scripts,function(script) {source(script)$value})
do.call(rbind,result)

  time.taken
1       0.87
2       0.51
3       0.61
4       0.38
5       0.37
6       0.91