Search code examples
arraysbashparallel-processingassociative-arraygnu-parallel

How to pass associative array to Gnu Parallel


Like in a title, I've declared bunch of variables and a function. And when I pass them through a function I've got what I expected. But running same code through parallel didn't... How to fix it?

#!/bin/bash                                                

declare -xA MAP # export associative array                 
declare -x str="ing" # export variable                     
MAP[bar]="baz"                                             
MAP[bar2]="baz2"                                           


foo() {                                                    
  echo "$@"                                                
  echo "variable: ${str}"                                  
  echo "map: ${MAP[@]}"                                    
}                                                          

export -f foo                                              

foo "call function directly:"          

call function directly:
variable: ing
map: baz2 baz

parallel foo ::: "call function through parallel" ::: 1 2 3

call function through parallel 1
variable: ing
map:
call function through parallel 2
variable: ing
map:
call function through parallel 3
variable: ing
map:

edit after comments

It looks that accepted answer for this question is: There isn't really a good way to encode an array variable into the environment.

Which is a bit sad... ;)


Solution

  • From comments under the question it looks that accepted answer is: There isn't really a good way to encode an array variable into the environment.

    And that's all... ;)

    It looks that there is another one: Accessing Associative Arrays in GNU Parallel with quite sophisticated approach. But...

    after some tinkering (as Socovi suggested that specific problem can have solutions) it looks that for my case good enough workaround is to "serialize" array in temporary file and deserialize it in function.

    #!/bin/bash                                                                   
    
    declare -A MAP # export associative array                                    
    declare -x str="ing" # export variable                                        
    MAP[bar]="baz"                                                                
    MAP[bar2]="baz2"                                                              
    declare -x serialized_array=$(mktemp)   
    
    # declare -p can be used to dump the definition 
    # of a variable as shell code ready to be interpreted                                      
    declare -p MAP > "${serialized_array}" 
    
    
    # perform cleanup after finishing script                                      
    cleanup() {                                                                   
      rm "${serialized_array}"                                                    
    }                                                                             
    trap cleanup EXIT                                                             
    
    
    foo() {                                                                       
      echo "$@"                                                                   
      echo "variable: ${str}"                                                     
      source "${serialized_array}" # deserialize an array                         
      echo "map: ${MAP[@]}"                                                       
    }                                                                             
    
    export -f foo                                                                 
    
    foo "call function directly:"  
    

    call function directly:
    variable: ing
    map: baz2 baz

    parallel foo ::: "call function through parallel" ::: 1 2 3 
    

    call function through parallel 1
    variable: ing
    map: baz2 baz
    call function through parallel 2
    variable: ing
    map: baz2 baz
    call function through parallel 3
    variable: ing
    map: baz2 baz