How can I avoid StackOverflow Error on Julia?

Question

This is a little product function that transform a list of constants, a list of operators and another list of constants to a Julia Expr. The function is working for smalls sets of constants, however for more a list of 100 constants and 5 operators, the function throws a StackOverflow.

function applyProduct(left, operators, right)
    prod = Iterators.product(left, operators, right)
    prod = Iterators.vcat(prod...)
    return Iterators.map(x -> Expr(:call, Symbol(x[2]), Symbol(x[1]), Symbol(x[3])), prod)
end

My function blows up at the vcat function:

ERROR: LoadError: StackOverflowError:
Stacktrace:
[1] promote_eltypeof(::Tuple{String,typeof(+),Expr}, ::Tuple{String,typeof(+),Expr}, ::Vararg{Any,N} where N) at .\abstractarray.jl:1211 (repeats 8406 times)
[2] _cat(::Val{1}, ::Tuple{String,typeof(*),Expr}, ::Vararg{Any,N} where N) at .\abstractarray.jl:1382
[3] #cat#104(::Val{1}, ::Function, ::Tuple{String,typeof(*),Expr}, ::Vararg{Any,N} where N) at .\abstractarray.jl:1511
[4] (::getfield(Base, Symbol("#kw##cat")))(::NamedTuple{(:dims,),Tuple{Val{1}}}, ::typeof(cat), ::Tuple{String,typeof(*),Expr}, ::Vararg{Any,N} where N) at .\none:0
[5] vcat(::Tuple{String,typeof(*),Expr}, ::Tuple{String,typeof(*),Expr}, ::Tuple{String,typeof(*),Expr}, ::Vararg{Any,N} where N) at .\abstractarray.jl:1449

It's a stackoverflow error so the data is somehow stored on the stack. Should I declare something to keep it in-memory ?

I would like to apply this operation to a large sets of constants. Maybe I'm not using the most efficient approach.

Answer 1

You definitely want to pre-allocate. And avoid explicitly enumerating the product iterator and using maps

function applyproduct(left, op, right)
    itr = Iterators.product(op, left, right)
    thestack = Vector{Expr}(undef, length(itr))

    for (i, t) in enumerate(itr)
        thestack[i] = Expr(:call, Symbol.(t)...)
    end

    return thestack
end

N = 500
left = [String(rand('a':'z', rand(1:5))) for _ in 1:N]
right = [String(rand('a':'z', rand(1:5))) for _ in 1:N]
op = ["+", "-", "/", "^", "<", "≤"]

applyproduct(left, op, right)
# 1500000-element Array{Expr,1}:
#  :(wtolz + kzyxh)
#  :(wtolz - kzyxh)
#  ⋮             

Also, you can avoid the Expr call with:

:($(Symbol(t[1]))($(Symbol(t[2])), $(Symbol(t[3]))))  
# :(adsf - sd)