Process pool in Bash

Question

I am trying to get a for loop that will only execute a threaded volatility yara scan module based off a list of rules. One scan per rule per core. Basically it should take the current number of vol.py processes running and check against the core count and maximize the processes running.

However, when I run this it will spawn a volatility process for every rule and not just the max of 4 (based off my core count). Any help with the syntax here would be great.

Currently I have this:

count=$(ps aux | grep vol.py | wc -l)
cores=$(cat /proc/cpuinfo | grep processor | wc -l)
for x in $(cat $rules_path/rules.txt)
do
  if [ $count -lt $cores ]; then # we are under the limit
     echo $x
     vol.py -f /home/cuckoo/.cuckoo/storage/analyses/12/memory.dmp --profile=Win7SP1x64 yarascan --yara-file=$rules_path/allrules/$x --output=text --output-file=$out_dir/$x.log &>/home/$name/Desktop/error.txt & 
  else
     wait
  fi
done

Answer 1

Your approach could work, with something like:

cores=$(cat /proc/cpuinfo | grep processor | wc -l)
count=$(ps aux | grep vol.py | wc -l)
for x in $(cat $rules_path/rules.txt)
do
  if [ $count -lt $cores ]; then # we are under the limit
     echo $x
     vol.py -f /home/cuckoo/.cuckoo/storage/analyses/12/memory.dmp --profile=Win7SP1x64 yarascan --yara-file=$rules_path/allrules/$x --output=text --output-file=$out_dir/$x.log &>/home/$name/Desktop/error.txt & 
     count=$(ps aux | grep vol.py | wc -l)
  else
     wait -n
  fi
done

All I've changed is:

• recalculate count each time we add a process
• wait -n -- to just wait for one job to end

---

However there are simpler ways to achieve this. One is with xargs --max-procs:

cat $rules_path/rules.txt | xargs --max-procs="$cores" -n1 call_volatility 

... where call_volatility is a script like this:

#!/bin/bash
x=$1
vol.py -f /home/cuckoo/.cuckoo/storage/analyses/12/memory.dmp \
     --profile=Win7SP1x64 yarascan \
     --yara-file=$rules_path/allrules/$x \
     --output=text \
     --output-file=$out_dir/$x.log \
     &>/home/$name/Desktop/error.txt

---

There is no guarantee with either approach that the processes will be evenly distributed between your cores.