Search code examples
phpcurlscreen-scraping

Problem with curl, xpath query


I need some help with my xpath query. I can get this code to work with just about every site I need to scrape except this small part of a particular site... I just get a blank page... Does anyone have an idea on how I can do this better?

//
$target_url = "http://www.teambuy.ca/vancouver/";
$userAgent = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)';

// make the cURL request to $target_url
$ch = curl_init();
curl_setopt($ch, CURLOPT_USERAGENT,$userAgent);
curl_setopt($ch, CURLOPT_URL,$target_url);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
$html= curl_exec($ch);
if (!$html) {
    echo "<br />cURL error number:" .curl_errno($ch);
    echo "<br />cURL error:" . curl_error($ch);
    exit;
}

// parse the html into a DOMDocument
$dom = new DOMDocument();
@$dom->loadHTML($html);

// grab all the on the page
$xpath = new DOMXPath($dom);
$hrefs = $xpath->evaluate("/html/body/div[@id='pagewrap']/div[@id='content']/div[@id='bottomSection']/div[@id='bottomRight']/div[@id='sideDeal']/div[2]/div/a/center/span");

foreach ($hrefs as $e) {
    $e->nodeValue;
}
$insert = $e->nodeValue;
echo "$insert";

--EDIT--

No luck... Fatal error: Call to a member function loadHTMLfile() on a non-object in ... Line 4 //

$xpath_query = $dom->loadHTMLfile("http://www.teambuy.ca/vancouver/");

$hrefs = $xpath_query->evaluate("/html/body/div[7]/div[4]/div[3]/div[2]/div[1]/div[2]/div/a/center/span");

foreach ($hrefs as $e) {
    echo $e->nodeValue;
}
$insert = $e->nodeValue;

echo "$insert";

Solution

  • don't use cURL. just use

    $dom->loadHTMLFile("http://www.teambuy.ca/calgary/");
    

    don't use

    $xpath = new DOMXPath($dom);
    

    just use

    $href = $dom->xpath($xpath_query);
    

    I imagine your xpath query could be simplified as well...

    also,

    foreach ($hrefs as $e) {
        $e->nodeValue;
    }
    

    does nothing. might want to try this instead.

    foreach ($hrefs as $e) {
        echo $e->nodeValue;
    }