How do you load a module @everywhere inside a function in Julia

Question

I am trying to load a module in my workers after creating them with addprocs. When addprocs is called at the top level, everything is fine. However, I am not able to do the same thing when I wrap the code in a function.

In my case, I am adding workers dynamically, so it is not feasible to call @everywhere using XXX always on the top level, I would need to do this inside a function.

In short, this works:

addprocs(1)
@everywhere using XXX

and this doesn't:

function myaddprocs()
    addprocs(1)
    @everywhere using XXX
end

Any ideas?

Answer 1

After a bit more investigation, I have pinpointed some of the issues which were making my code not work.

1. Imports must happen after addprocs. If an import has happened before, the import must be prefixed by an @everywhere.

2. Top-level expressions (such as using) inside functions do not work, unless wrapped in an eval statement.

A fix to my code would be:

function myaddprocs()
    addprocs(1)
    eval(macroexpand(quote @everywhere using XXX end))
end

---

Examples

I have tested the following snippets on Julia 0.6.1. I have also tested them using the same version on a SGE cluster (OGS/GE 2011.11p1), by substituting all addprocs by addprocs_sge, and importing ClusterManagers.jl. The following snippets work:

• using after addprocs:

  addprocs(1)
  using SpecialFunctions
  pmap(x->SpecialFunctions.sinint(1), workers())
  

• using before and after addprocs, the second with @everywhere:

  using SpecialFunctions
  addprocs(1)
  @everywhere using SpecialFunctions
  pmap(x->sinint(1), workers())
  

• using wrapped in eval after addprocs within function

  function getprocs()
      addprocs(1)
      eval(Expr(:using,:SpecialFunctions))
      pmap(x->SpecialFunctions.sinint(1), workers())
  end
  getprocs()
  

• Same as before with with @everywhere applied to eval

  function getprocs()
      addprocs(1)
      @everywhere eval(Expr(:using,:SpecialFunctions))
      pmap(x->sinint(1), workers())
  end
  getprocs()
  

• Same as before with with @everywhere within eval instead

  function getprocs()
      addprocs(1)
      eval(macroexpand(quote @everywhere using SpecialFunctions end))
      pmap(x->sinint(1), workers())
  end
  getprocs()
  

These snippets, on the other hand, do not work:

• using before addprocs

  using SpecialFunctions
  addprocs(1)
  pmap(x->SpecialFunctions.sinint(1), workers())
  

• using before and after addprocs

  using SpecialFunctions
  addprocs(1)
  using SpecialFunctions
  pmap(x->SpecialFunctions.sinint(1), workers())
  

• using within function

  using SpecialFunctions
  function getprocs()
      addprocs(1)
      @everywhere using SpecialFunctions
      pmap(x->sinint(1), workers())
  end
  getprocs()
  

• using within function

  function getprocs()
      addprocs(1)
      using SpecialFunctions
      pmap(x->SpecialFunctions.sinint(1), workers())
  end
  getprocs()