Search code examples
phpasynchronouscurlparallel-processingcurl-multi

Asynchronous/parallel HTTP requests using PHP curl_multi


I recently looked into the possibility of making multiple requests with curl. I may not be understanding it fully, so I am just hoping to clarify some concepts.

It's definitely a good option if you are fetching content from multiple sources. That way, you can start processing the results from faster servers while still waiting for slower ones. Does it still make sense to use it if you are requesting multiple pages from the same server? Would the server still serve multiple pages at the time to the same client?


Solution

  • You can't do multi-threading in PHP, so you won't be able to start processing one page while the others are still being retrieve. Multi-curl won't return control until all pages are retrieved or timeout. So it will take as long the it takes for the slowest page to be retrieved. You are going from serial (curl) to parallel (multi_curl), which will still give you a big boost.

    Servers will serve multiple pages to the same client up to a certain configure limit. Requesting 5-10 pages from a server would be fine.