Search code examples
phpsymfonyweb-scrapingguzzlegoutte

Unable to scrape Google

I'm trying to scrape google for reverse image search results using Goutte (its basically a wrapper around Guzzle + Symfony DOM parser).

I'm doing:

$googleURL = "https://www.google.com/searchbyimage?&image_url=".$imageURL;

$crawler = $client->request('GET', $googleURL);
$client->followRedirects(true);
print_r($crawler,1);

.. and the output (https://paste.ee/p/spfYA) contains the correct URL but the body (HTML) is of the default search page, the one without results.

What should I do to get the search results as HTML.

Solution

  • All I had to do was set the user-agent:

    $client->setHeader('User-Agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.111 Safari/537.36');