Stress test API using multiple JSON files

Question

I am trying to fire 40000 requests towards an API using 40000 different JSON files.

Normally I could do something like this:

for file in /dir/*.json
do
#ab -p $file -T application/json -c1 -n1 <url>
curl -X POST -d@"$file" <url> -H "Content-Type: application/json"
done;

My problem is that I want to run simultaneous requests, e.g. 100 and I want the total time it took to send all requests etc. recorded. I can't use the -c 100 -n 40000 in ab since its the same URL with different files.

The files/requests all look something like

{"source":"000000000000","type":"A"}
{"source":"000000000001","type":"A"}
{"source":"000000000003","type":"A"}

I was not able to find any tool that supports this out of the box (e.g. Apache Benchmark - ab).

I came across this example here on SO (modded for this question). Not sure I understand why that example would "cat /tmp" when mkfifo tmp is a file and not a dir though. Might work?

mkfifo tmp
counter=0
for file in /dir/*.json
do
    if [ $counter -lt 100 ]; then
        curl -X POST -H "Content-Type: application/json" -d@"$file" <url> &
        let $[counter++];
    else
        read x < tmp
        curl -X POST -H "Content-Type: application/json" -d@"$file" <url> &
    fi
done;
cat /tmp > /dev/null
rm tmp

How should I go about achieving this in perl, ksh, bash or similar or does anyone know any tools that supports this out of the box?

Thanks!

Answer 1

If your request is just to time the total time take for sending these 40000 curl requests with different JSON each time, you can use good use of GNU parallel. The tool has great ways achieve job concurrency by making use of multiple cores on your machine.

The download procedure is quite simple. Follow How to install GNU parallel (noarc.rpm) on CentOS 7 for quick and easy list of steps. The tool has a lot more complicated flags to solve multiple use-cases. For your requirement though, just go the folder containing these JSON files and do

parallel --dry-run -j10 curl -X POST -H "Content-Type: application/json" -d@{} <url> ::: *.json

The above command tries to dry run your command, in terms of how parallel sets up the flags and processes its arguments and starts running your command. Here {} represents your JSON file. We've specified here to run 10 jobs at a time and increase the number depending on how fast it runs on your machine and by checking the number of cores on your machine. There are also flags to limit the overall CPU to be allowed use by parallel, so that it doesn't totally choke your system.

Remove --dry-run to run your actual command. And to clock the time taken for the process to complete, use the time command just prefix it before the actual command as time parallel ...