Search code examples
environment-variablesjuliahpcpbs

Set environment variables all over processes in Julia

I'm currently working with Julia (1.0) to run some parallel code on clusters of an HPC. The HPC is managed with PBS. I'm trying to find a way for broadcasting environment variables over all processes, i.e. a way to broadcast a specific list of environment variables automatically in order to have access to them in every Julia worker.

#!/bin/bash
#PBS ...

export TOTO=toto
julia --machine-file=$PBS_NODEFILE my_script.jl

In this example, I will not be able to access to the variable TOTO in each julia worker (via ENV["TOTO"]).

The only way I found to do what I want is to set the variables in my .bashrc but I want this to be script-specific. Another way is to put in my startup.jl file :

@everywhere ENV["TOTO"] = $(ENV["TOTO"])

But it is not script-specific because I have to know in advance which variables I want to send. If I do a loop over ENV keys then I'll broadcast all the variables and then override variables I don't want to.

I tried to use DotEnv.jl but it doesn't work.

Thanks for your time.

Solution

  • The obvious way is to set the variables first thing in script.jl. You can also put the initialization in a separate file, e.g. environment.jl, and load that on all processes with the -L flag:

    julia --machine-file=$PBS_NODEFILE -L environment.jl my_script.jl

    where environment.jl would, in this case, contain

    ENV["TOTO"] = "toto"

    etc.