I want to use r packages on cran such as forecast etc with sparkr and meet following two problems.
Should I pre-install all those packages on worker nodes? But when I read the source code of spark this file, it seems that spark will automatically zip packages and distribute them to the workers via --jars or --packages. What should I do to make the dependencies available on workers?
Suppose I need to use functions provided by forecast in a map transformation, how should I import the package. Do I need to do something like following, import the package in the map function, will it make multiple import:
SparkR:::map(rdd, function(x){
library(forecast)
then do other staffs
})
Update:
After reading more source code, it seems that, I can use includePackage to include packages on worker nodes according to this file. So now the problem becomes is it right that I have to pre-install the packages on nodes manually? And if that's true, what's the use case for --jars and --packages described in question 1? If that's wrong, how to use --jars and --packages to install the packages?
It is boring to repeat this but you shouldn't use internal RDD API in the first place. It's been removed in the first official SparkR release and it is simply not suitable for general usage.
Until new low level API* is ready (see for example SPARK-12922 SPARK-12919, SPARK-12792) I wouldn't consider Spark as a platform for running plain R code. Even when it changes adding native (Java / Scala) code with R wrappers can be a better choice.
That being said lets start with your question:
RPackageUtils are designed to handle packages create with Spark Packages in mind. It doesn't handle standard R libraries.
Yes, you need packages to be installed on every node. From includePackage docstring:
The package is assumed to be installed on every node in the Spark cluster.
* If you use Spark 2.0+ you can use dapply, gapply and lapply functions.