Search code examples
parallel-processingjuliadistributed-computing

Julia - Transferring data between workers


Let's say that we have defined two workers in Julia. I am wondering if it is possible to send data computed in worker 3, directly to worker 2 without fetching data in master process and sending it to worker 2, afterwards? In general, with the existing parallel tools in Julia i.e. @spawn, fetch and etc, is it possible that one worker gets data (messages) from different workers and use them in order to compute a message for another worker?

Note: I know that there is a MPI wrapper which may be more compatible with this issue, however still I am wondering if this can be done with the existing parallel tools in Julia? In other words, when they say that message passing in Julia is “one-sided”, does it mean that transferring data between workers is not feasible?


Solution

  • Using ParallelDataTransfer.jl, just do

    # Get an object from named x from Main module on process 2. Name it y
    y = @getfrom 2 x
    

    and see the README for more examples.