I like GNU Parallel and have tried to use it for pagination but need help to get it working successfully. Basically, I am following the use cases on the Quickblox API guide to get data:
http://quickblox.com/developers/Custom_Objects#Get_related_records
The maximum number of records one can retrieve is 100 per page, and one can only retrieve a page at a time. These are specified via the -d parameter. I want to use GNU Parallel to obtain pages 1..79.
I found a thread that explains how to use GNU Parallel when you have parameters that take on many different values but haven't been able to successfully adapt it to my case.
GNU Parallel - parallelize serial command line programs without changing them
Your help would be greatly appreciated!
curl -X GET -H "QB-Token: 7de49c25f44e557aeed1b635" -d "page=3" -d "per_page=100" https://api.quickblox.com/users.xml > qblox_users_page3_100perpage
If you want output in different files:
parallel 'curl -X GET -H "QB-Token: 7de49c25f44e557aeed1b635" -d "page={}" -d "per_page=100" https://api.quickblox.com/users.xml > qblox_users_page{}_100perpage' ::: {1..79}
If you want it in a single big file:
parallel -k 'curl -X GET -H "QB-Token: 7de49c25f44e557aeed1b635" -d "page={}" -d "per_page=100" https://api.quickblox.com/users.xml' ::: {1..79} > qblox_users