Does anyone have experience using C extensions to Mathematica (LibraryLink or MathLink -- currently I'm using LibraryLink) with remote parallel kernels?
In short: How can I transparently use a LibraryLink-defined function (see CreateLibrary and LibraryFunctionLoad) in both parallel and non-parallel evaluations when the subkernels run on a remote machine?
I am looking for some setup steps that will allow me to have a libraryFun function (written in C) that can be called either normally as libraryFun[args], or in parallel (both of Parallelize@Table[libraryFun[arg], {arg, 0, 100}] and the same with ParallelTable[]) when the subkernels run on a remote machine.
Running the main kernel remotely too might be better if I weren't having trouble with that as well.
In the meantime I made some progress. I'll describe it here.
First, ParallelEvaluate will evaluate an expression in all parallel kernels. If the source files for the C extension are copied to the remote machine, we can compile them there like this:
ParallelNeeds["CCompilerDriver`"]
k1 = First@Kernels[]
ParallelEvaluate[SetDirectory["/path/to/source/files"]]
ParallelEvaluate[CreateLibrary["sourefile", "myLibrary"]]
This needs to be done one time only. I assume that the library has been already compiled on the main kernel machine.
After this, in all subsequent sessions we can use FindLibrary on both the main and the remote machines to load the library.
LibraryFunctionLoad[myFun = FindLibrary["myLibrary"], "fun", ...]
ParallelEvaluate[myFun = LibraryFunctionLoad[FindLibrary["myLibrary"], "fun", ...]]
And here comes the trouble. Because of different paths, myFun will have different values in the main and in the parallel kernels.
So the question is: How can ensure that the value of myFun will not accidentally get synchronized between the main and the parallel kernels?
I'll show in an isolated examples how this might accidentally happen:
In[1]:= LaunchKernels[2]
Out[1]= {KernelObject[1, "local"], KernelObject[2, "local"]}
Set value of x in main kernel:
In[2]:= x = 1
Out[2]= 1
Note that it gets the same value in remote kernels too:
In[3]:= ParallelEvaluate[x]
Out[3]= {1, 1}
Set a different value for x in the parallel kernels and verify that they keep it:
In[4]:= ParallelEvaluate[x = 2]
Out[4]= {2, 2}
In[5]:= {x, ParallelEvaluate[x]}
Out[5]= {1, {2, 2}}
Now "innocently" use Parallelize on something containing x:
In[6]:= Parallelize[Table[x, {10}]]
Out[6]= {1, 1, 1, 1, 1, 1, 1, 1, 1, 1}
And see how the value of x got re-synced between the main and subkernels.
In[7]:= {x, ParallelEvaluate[x]}
Out[7]= {1, {1, 1}}
The new question is: How can I prevent a certain symbol from ever auto-syncing between the main and the subkernels?
I seem to have found a solution to my question in Update above. It seems to work but I cannot confirm yet that it is not fragile.
The solution is to put symbols that we don't want synchronized into a separate context. Using c`x in my example in place of x prevents synchronizing the value of x when it is used inside Parallelize. Then we can add this context to the $ContextPath to make the symbol easily accessible.
The most convenient way to do this is probably putting all definitions in a package that loads the library functions using LibraryFunctionLoad[FindLibrary[...], ...]. For this to work, the library must have been compiled manually first on both the local and remote machine, however, the package code can be the exactly same both for the main and the subkernels.
I am still interested if someone can confirm that variables not in $Context are guaranteed not to be auto-synchronized.
Update It has been confirmed here.