run a bash for loop in parallel

Question

I've got this script that does a credential lookup for each host, in an on-premise vault system, and then runs an ansible-playbook for it.

#!/bin/bash

for host in `cat ~/.ansible/hosts`
  do
    SECRET=`/opt/vault/bin/get-admin-credential --tag=$host`
    HOST=`echo $SECRET | cut -d ';' -f1`
    LOGIN=`echo $SECRET | cut -d ';' -f2`
    DOMAIN=`echo $SECRET | cut -d ';' -f3`
    PWD=`echo $SECRET | cut -d ';' -f4`

    if [ -z "$DOMAIN" ]; then
      ansible-playbook -i ~/.ansible/hosts ~/.ansible/windows.yml -e "ansible_host=$HOST ansible_user=$LOGIN ansible_password=$PWD" --limit $host
    else
      ansible-playbook -i ~/.ansible/hosts ~/.ansible/windows.yml -e "ansible_host=$HOST ansible_user=$LOGIN@$DOMAIN ansible_password=$PWD" --limit $host
    fi
  done

This loops over each host sequentially, I've tried stuff with GNU parallel but haven't been able to do what I want, running the for loop with 5 in parallel.

Anyone point me in the right direction?

Answer 1

I don't have any "ansibles" or "vaults", so this is completely untested but may get you close:

doit(){
   host="$1"

   SECRET=$(/opt/vault/bin/get-admin-credential --tag=$host)
   HOST=$(echo $SECRET | cut -d ';' -f1)
   LOGIN=$(echo $SECRET | cut -d ';' -f2)
   DOMAIN=$(echo $SECRET | cut -d ';' -f3)
   PWD=$(echo $SECRET | cut -d ';' -f4)

   if [ -z "$DOMAIN" ]; then
      ansible-playbook -i ~/.ansible/hosts ~/.ansible/windows.yml -e "ansible_host=$HOST ansible_user=$LOGIN ansible_password=$PWD" --limit $host
   else
      ansible-playbook -i ~/.ansible/hosts ~/.ansible/windows.yml -e "ansible_host=$HOST ansible_user=$LOGIN@$DOMAIN ansible_password=$PWD" --limit $host
   fi
}

# Export doit function to subshells created by GNU Parallel
export -f doit

parallel -a ~/.ansible/hosts doit

Stylistically, there are maybe a few improvements. Firstly, shell variables consisting of upper case letters are reserved, so you shouldn't maybe use HOST, DOMAIN etc. Also, you can probably simplify all that unsightly cutting and echoing to extract the variables from the SECRET by using an IFS=';' and a read like this:

SECRET=$(/opt/vault/bin/get-admin-credential --tag=$host)
IFS=';' read host login domain pwd <<< "$SECRET"

So, my best and final answer is:

doit(){
   host="$1"

   secret=$(/opt/vault/bin/get-admin-credential --tag=$host)
   IFS=';' read host login domain pwd <<< "$secret"

   if [ -z "$domain" ]; then
      ansible-playbook -i ~/.ansible/hosts ~/.ansible/windows.yml -e "ansible_host=$host ansible_user=$login ansible_password=$pwd" --limit $host
   else
      ansible-playbook -i ~/.ansible/hosts ~/.ansible/windows.yml -e "ansible_host=$host ansible_user=$login@$domain ansible_password=$pwd" --limit $host
   fi
}

# Export doit function to subshells created by GNU Parallel
export -f doit

parallel -a ~/.ansible/hosts doit