Search code examples
phpcurlcurl-multi

Multiple and Growing Outbound CURL Request

Multi Outbound Request - Curl

Here's the problem, I have several clients I have to CURL outbound requests to. Say for instance I currently have 20 clients and I send around 100~1000 requests per minute to each of them. I also use CURL multi however it seems to have a limitation of how many requests it can make at a time and it will also depend on the longest CURL response for the entire routine to complete.

For instance I have the following clients:

Client 1
url: http://www.exampleclient1.com/process/

Client 2
url: http://www.exampleclient2.com/process/

... and so on

The main issue here is that I have a single script that does the job for every client. Say,

http://localhost/app/send/client1
> will send out the pending outbound queue to client 1's url 

http://localhost/app/send/client2 
> will send out the pending outbound queue to client 2's url

... and so on

The reason why I separated them is because there should be dedicated connections between clients and their latencies are different from each other. Some clients respond faster and have faster servers while some clients have slow servers or take more hops to reach.

Here's my question: Is there a means to simplify this process? Because it's a hassle that everytime I have to add a client to my database I'll also have to add 

http://localhost/send/newclient1 
http://localhost/send/newclient2
.
.
.
http://localhost/send/newclientn

to the list of cronjobs. Is it possible to put it in a single script instead so as my list of clients grow it will not affect the overall performance of the outbound CURL function I have?

By the way, I'm using PHP, CURL. If there's a solution that recommends the use of another technology other than PHP for this, a linux queuing manager...etc, you're welcome to suggest.

Thanks!

Solution

  • I have similar problem that you face, common issues are: - Different response time of Resources URLs ( as you may call them clients ) - The Demand of adding new URL you add one or ten at once.

    In addition to commons, my other observations:- - URL from different services providers, - with different data format - and they require different type of requests for what you want.

    I have reviewed couple of PHP-cURL libraries:

    Now managing the the database and clients' send requests, you will at least well know RDBMS such MySQL or SQLite, other options RDBMS are considered.

    • In the RDBMS create a table that contains requests as in { "clientNo" => "URL" }
    • You can add your list of clients using PHPmyAdmin or any database management interface.
    • The Script: create a script that fetch the whole entry of database, one by one within foreach loop; preform creation on CURL node and adding handler to queue of multiple CURl calls.
    • test your script, and for production add it to cronjob.
    • every time you have to add a client , just add him to RDBMS (MySQL, SQLite ) using database management interface.

    NOTE: if you are doing outbound management, you have to consider security issues, proxy; and possible create PHP and HTML form for submitting new clients.