Search code examples
rparallel-processingbayesianmcmc

Running Multiple Bayesian Chains in Parallel R:3 nodes produced errors; first error: incorrect number of dimensions

I am trying to run replicate chains of a Bayesian statistical function on multiple cores; 1 chain per core. The function mcmc below is a script to run a single mcmc chain. My thinking is I can just run the mcmc function three times, each instance on a separate core. I found a couple examples I tried to modify, but have been unable to get it to run appropriately. I get the following error: 3 nodes produced errors; first error: incorrect number of dimensions. This makes me think I am not understanding how to use the parallel version of the apply function. I keep thinking it should be straight forward but can't seem to find my error. I am learning so much about Bayesian statistics, programming, and computers on the fly. Can anyone tell me what I am doing wrong?

My apologies if this has been answered previously, I was unable to find an answer that helped.

library(parallel) 
library(snowfall) 
library(rlecuyer)

cps=detectCores()-5 #I have access to 8 cores, but want to target only three
sfInit(parallel=TRUE, cpus=cps)
sfExportAll()
sfClusterSetupRNG()

#necessary input; GB, all.layers, ind defined previously
nchain=3
n.mcmc=2000
df=9

#mcmc is a function to run a single mcmc chain
tmp.fcn <- function(i){
  tmp.out[i]=mcmc(GB,all.layers,ind,df,n.mcmc)
}


sfExport("GB","all.layers","ind","df","n.mcmc","nchain")
tmp.time=Sys.time()
score.list=sfClusterApplySR(1:nchain,tmp.fcn) 
time.1=Sys.time()-tmp.time
Solution

  • I figured out what my issues were. First, I switched packages and used parallel instead of snowfall. Second, I figured out I needed a list which breaks up my problem into pieces to apply my function to. In my case, instead of having the starting values for my variable t randomly generated inside the function I wanted to apply, I made a list of 3 random start values then applied my function (tmp.func) to that list (t.start). 

    library(parallel)
n.core<-detectCores()-5
cl <- makeCluster(n.core)

#write function
tmp.func<-function(i){
tmp.out=mcmc(GB,all.layers,ind,df=8,cov.1=.5,cov.0=.15,t.start[i],n.mcmc=11000)
tmp.out
}

t.start=c(.15,.1,.2)
clusterEvalQ(cl,{library(adegenet);library(Matrix);library(MASS);library(mvtnorm);library(fda);library(raster);library(rgdal)})

clusterExport(cl,c("t.start","GB","all.layers","ind","mcmc"))

#apply function
out.11k=parLapply(cl,1:length(t.start),tmp.func)


stopCluster(cl)