Search code examples
proxyweb-scrapinghttp-proxy

Using HTTP proxy in domCrawler component of Symfony


I'm using DomCrawler component to make DOM navigation for HTML and XML documents.

When I use like that:

use Goutte\Client;

$client = new Client();
$crawler = $client->request('GET', 'https://www.google.com');

I'll use my client IP to connect to google. Is there a way to use a proxy server for that connection to scrape HTML data masking my IP?


Solution

  • I was able to do that way:

    $client = new Client();
        $client->getClient()->setDefaultOption('config', ['curl' => [
            CURLOPT_PROXY => "proxyaddress:proxy",
            CURLOPT_PROXYUSERPWD => "user:pass",
            CURLOPT_COOKIEFILE => dirname(__FILE__). "/cookies.txt",
            CURLOPT_COOKIEJAR => dirname(__FILE__). "/cookies.txt"
        ]
        ]);