Search code examples
phpsymfonyhttprequestguzzle

How to put a delay between package of HTTP requests with Guzzle PHP?


I build an API with Symfony. During an action, the data coming from the front are website links and I use them to create and send asynchronous HTTP GET requests simultaneously (using of the Scrapestack API who scrapes these websites). But the fact is that the number of website links can be large and can be on the same domain. In order not to be blocked by a domain I would like to put a delay of 1sec between package of 10 requests sended simultaneously. Is it possible to do this with the PHP HTTP client Guzzle (https://github.com/guzzle/guzzle) ? Do I have to use Pool ? Here is the actual code :

$promises = [];
$results = [];
foreach ($data as $d){
   if(gettype($d) === 'string'){
     $d = json_decode($d, true);
   }
   $url = sprintf('%s?%s', 'http://api.scrapestack.com/scrape', $this->createScrapestackRequestData($d['link']));
   array_push($promises, $this->client->getAsync($url));
}
$responses = Utils::settle($promises)->wait();

Solution

  • SOLUTION :

    $requests = [];
    $results = [];
    foreach ($data as $d){
      if(gettype($d) === 'string'){
        $d = json_decode($d, true);
      }
      array_push($requests, $this->curlClient->request('GET', $this->getUrlScrapestackApi($d['link'])));
    }
    foreach ($requests as $index => $response) {
      if ($index !== 0 && $index % 10 === 0) {
        sleep(1);
      }
      array_push($responses, $response->getContent());
    }
    

    Info : $this->curlClient is an instance of Symfony\Component\HttpClient\CurlHttpClient/