Search code examples
pythonrforeachjupyter-notebookparallel-processing

R foreach runs forever jupyter notebook with rpy2 python


I have a python jupyter notebook where I run some data-getting code which gives me a list of fcs files. I use fcsparser in R with a foreach in parallel to parse these like so:

%%R -i files -o data_fcs
library(dplyr)
library(doParallel)
library(foreach)
library(flowCore)
registerDoParallel(cores=4)
trans = arcsinhTransform(transformationId='arcsinhTransform',a=0,b=(1/5),c=0)
print("a")
data_fcs <- foreach(file=files) %dopar% {
    library(flowCore)
    print(sprintf("%s", file))
    # load fcs file
    fcs = read.FCS(file)........(more code after this)

The "a" is printed before the foreach loop but the stuff inside never is executed I guess because nothing else is printed as the code just runs endlessly. My machine (MacBook pro laptop) has 4 cores.

Sometimes, the code randomly works in about a minute and the script finishes, which is very odd to me. What could be the cause of this issue? Thanks!


Solution

  • I found a "solution" which is just to restart jupyter after each successful run...seems to be working now