GNU parallel inheriting environment variable while executing a local script

Question

Suppose I have foo.sh that calls bar.sh using parallel:

# foo.sh

#! /bin/bash

parallel -N 3 bar.sh ::: $(seq 10)

My bar.sh works like this: if there is an environment variable (e.g. DEBUG=1) set, then it will output lots of debug info.

Ideally I want to simply execute my foo.sh like such:

$ DEBUG=1 foo.sh

Normally, foo.sh has $DEBUG value, as well as bar.sh sees it. But now with me using GNU parallel to call bar.sh, which is a local program, my bar.sh no longer has the DEBUG value set.

I read that --env only works if I had remote execution -S set, and from me trying it does not seem to work for me.

Is there a way to get my parallel'ed bar.sh to simply "inherit" the environment settings of my foo.sh? I really don't want to spell out each and every environment variable and their values when calling bar.sh in parallel.

TIA

Answer 1

You are looking for env_parallel which does exactly this.

Put this in $HOME/.bashrc:

  . `which env_parallel.bash`

E.g. by doing:

  echo '. `which env_parallel.bash`' >> $HOME/.bashrc

aliases

  alias myecho='echo aliases'
  env_parallel myecho ::: work
  env_parallel -S server myecho ::: work
  env_parallel --env myecho myecho ::: work
  env_parallel --env myecho -S server myecho ::: work

functions

  myfunc() { echo functions $*; }
  env_parallel myfunc ::: work
  env_parallel -S server myfunc ::: work
  env_parallel --env myfunc myfunc ::: work
  env_parallel --env myfunc -S server myfunc ::: work

variables

  myvar=variables
  env_parallel echo '$myvar' ::: work
  env_parallel -S server echo '$myvar' ::: work
  env_parallel --env myvar echo '$myvar' ::: work
  env_parallel --env myvar -S server echo '$myvar' ::: work

arrays

  myarray=(arrays work, too)
  env_parallel -k echo '${myarray[{}]}' ::: 0 1 2
  env_parallel -k -S server echo '${myarray[{}]}' ::: 0 1 2
  env_parallel -k --env myarray echo '${myarray[{}]}' ::: 0 1 2
  env_parallel -k --env myarray -S server echo '${myarray[{}]}' ::: 0 1 2

env_parallel is part of GNU Parallel 20160722. It is beta quality, so please report bugs if you find any.

If your know your UNIX you will know that you cannot use aliases, non-exported functions, non-exported variables, and non-exported arrays in shells started from the current shell (e.g. in bash -c); and especially not if the shell is remote (e.g. ssh server myalias). With env_parallel this common knowledge has to be revised into: you cannot do it without cheating.