Search code examples
parallel-processingjuliaincludecpuusing

Parallelism of Using and include, or at least, faster ways?

I am starting to have a big project and I am currently using and including many of packages and .jl files:

a = time()
@info "Loading JuMP"
using JuMP
@info "Loading Gurobi"
using Gurobi
# @info "Loading Combinatorics, DelimitedFiles, Dates and StatsBase"
# using Combinatorics, DelimitedFiles, Dates, StatsBase
@info "Loading Combinatorics, DelimitedFiles, Dates and Random"
using Combinatorics, DelimitedFiles, Dates, Random
@info "Loading Distributions, Graphs, GraphsFlows[TODO] and Plots"
using Distributions
# using Graphs, GraphsFlows, GraphPlot, Plots
using Graphs, GraphPlot, Plots
@info "Loading Parameters and Formatting"
using Parameters, Formatting #https://stackoverflow.com/a/58022378/10094437
@info "Loading Compose, Cairo and Fontconfig"
using Cairo, Fontconfig, Compose
@info "Loading .jl files $(lpad("0%",4))"

include("write_tikz.jl")
include("with_kw_mutable_structures.jl")
include("instance.jl")
include("solution_checker.jl")
@info "Loading .jl files $(lpad("25%",4))"
include("create_subtour_constraint.jl")
# include("ilp_rho_rsp_st_chains.jl")
include("ilp_rho_rsp_without_uc.jl")
include("benders_rho_rsp.jl")
include("benders_subproblem_poly.jl")
@info "Loading .jl files $(lpad("50%",4))"
include("benders_subproblem_ilp_primal.jl")
include("benders_subproblem_ilp_dual.jl")
include("print.jl")
include("three_four_rho_rsp.jl")
@info "Loading .jl files $(lpad("75%",4))"
include("utilities.jl")
include("rho_rsp_lb.jl")
include("./plots/plots.jl")
include("local_searches.jl")
@info "Loading .jl files $(lpad("100%",4))"
@info time()-a

All these using and includes take 34 seconds each time a launch Julia. It is indeed faster when already compiled, and when doing include("main.jl") for the second time, but still it takes 2s45 when already compiled once.

I would like to know if there are faster ways to using packages and includes Julia files, maybe with parallelism?

I am using Julia 1.7.2

Solution

  • These are the steps to consider

    1. Wrap all your code into a Julia package. The Julia packages will be precompiled on the first use and the subsequent times will be much shorter. You do not want to have a code with so many includes - make a package to speed up the things.
    2. Most likely you will be fine after step (1). However the next option is compiling the packages into Julia system image using PackageCompiler.
    3. As mentioned by @jing each new Julia version will do the job in a shorter time