Create sparse matrix in parallel in Julia

Question

I am trying to parallelize the creation of a sparse matrix in Julia. Inspired by this post this post I am trying this:

using Distributed
addprocs(4)
@everywhere using DistributedArrays

rows = [Int[] for _ in procs()]
cols = [Int[] for _ in procs()]
vals = [Float64[] for _ in procs()]
distribute(rows)
distribute(cols)
distribute(vals)

@sync @distributed for i = 1:1000
    for j = 1:1000
        v = exp(-(i - j)^2)
        if v > 0.1
            push!(localpart(rows)[1], i)
            push!(localpart(cols)[1], j)
            push!(localpart(vals)[1], v)
        end
    end
end

ROWS = vcat(rows...)
COLS = vcat(cols...)
VALS = vcat(vals...)

K = sparse(ROWS, COLS, VALS)
# K = 0×0 SparseMatrixCSC{Float64, Int64} with 0 stored entries

This outputs an empty matrix, and it does not get filled. But I found that if I call @fetchfrom 2 rows, the rows that it creates is not empty. So it seems that it is just not combining everything.

How can I fix this?

Answer 1

distribute(rows) does not modify rows to be distributed; it returns a new distributed array filled with the input. You have to work with its result, something like

rows = distribute([Int[] for _ in procs()])