How to invoke while statement multiple times in parallel using xargs max-procs utility

Question

I have a file called test.txt in Linux. The contents are below.

test.txt

['table1']
['table2']
['table3']
['table4']
['table5']
and so on

Now I have a while statement that loops over the test.txt file does some jobs.

While statement is below:

while read -r line; do
  table=${line:2:-2}
  validateTable=$(hive --database history -e "SHOW TABLES LIKE '$table'")
  if [[ -z $validateTable ]]; then
      /home/"$USER"/import.py "${table}"
  else
       /home/"$USER"/append.py "${table}"
  fi
done < < test.txt

In this while statement validateTable is to check whether table is present in hive or not.

If not present then it will invoke import.py script

if present it will invoke append.py script.

Now the while statement works fine. I am getting the expected result.

Requirement:

What I require is this I want to invoke the while statement in parallel. I mean I want the while to run 10 times at the same time.

What is the best possible solution.

I found that we can do it with xargs --max-procs option but cannot figure out how to use it.

Answer 1

xargs requires a single command to run, which means you would need to put the body of the loop into a shell script. Something like (call it myscript)

#!/bin/bash
table=${1:2:-2}
validateTable=$(hive --database history -e "SHOW TABLES LIKE '$table'")
if [[ -z $validateTable ]]; then
  /home/"$USER"/import.py "${table}"
else
  /home/"$USER"/append.py "${table}"
fi

Then run something like

xargs --max-procs 10 myscript < test.txt