Search code examples
phpscreen-scraping

Php scraping website content and show result with comma seperator


I have this script of scraping from website through PHP. All I want is to show result with comma separator and there are so many pages with paging. Just to show them with.

my code is

    $ch = curl_init('http://www.qatarliving.com/v3/classifieds/search/category/mobile-devices');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

/* 
 * XXX: This is not a "fix" for your problem, this is a work-around.  You 
 * should fix your local CAs 
 */
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);

/* Set a browser UA so that we aren't told to update */
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36');

$res = curl_exec($ch);

if ($res === false) {
    die('error: ' . curl_error($ch));
}

curl_close($ch);

$d = new DOMDocument();
@$d->loadHTML($res);

$output = array(
    'class' => '',
);

$x = new DOMXPath($d);



$myspan = $x->query('//span[@class="b-card b-card-mod-h item  "]');
if($myspan->length > 0){
    foreach($myspan as $row){
        echo $row->nodeValue . "<br/>";
    }
}

and result is

2,000 QAR Mobile phones, Al Gharrafa iPhone 6 128 like new By professional76
75 QAR Mobile phones, Other Virtual Reality Cardboard [NEW] By 1StopGulf

...


Solution

  • try below solution $data_array will contains desired output array:

    <?php
    $ch = curl_init('http://www.qatarliving.com/v3/classifieds/search/category/mobile-devices');
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    
    /*
     * XXX: This is not a "fix" for your problem, this is a work-around.  You
     * should fix your local CAs
     */
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
    
    /* Set a browser UA so that we aren't told to update */
    curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36');
    
    $res = curl_exec($ch);
    
    if ($res === false) {
        die('error: ' . curl_error($ch));
    }
    
    curl_close($ch);
    
    $d = new DOMDocument();
    @$d->loadHTML($res);
    
    $output = array(
        'class' => '',
    );
    
    $x = new DOMXPath($d);
    
    $myspan = $x->query('//span[@class="b-card b-card-mod-h item  "]');
    $data_array = array();
    if ($myspan->length > 0) {
        foreach ($myspan as $row) {
            $data = $row->getElementsByTagName('p');
            $array = array();
            foreach ($data as $dt) {
                $tag = '';
                $class = $dt->getAttribute('class');
                $value = $dt->nodeValue;
                if ($class == 'b-card--el-deposit-val') {
                    $tag = 'price';
                } else if ($class == 'b-card--el-deposit-time') {
                    $tag = 'deposittime';
                } else if ($class == 'b-ad-excerpt b-par-mod-clear b-line-mod-thin--mix-item') {
                    $tag = 'category';
                } else if ($class == 'b-card--el-description') {
                    $tag = 'name';
                }
                if ($tag) {
                    $array[$tag] = $value;
                }
            }
    
            $data = $row->getElementsByTagName('a');
            foreach ($data as $dt) {
                $tag = '';
                $class = $dt->getAttribute('class');
                $value = $dt->nodeValue;
                if (trim($class) == 'b-card--el-agency-title') {
                    $tag = 'addedby';
                }
                if ($tag) {
                    $array[$tag] = $value;
                }
            }
            $data_array[] = $array;
        }
        echo '<pre>';
        print_r($data_array);
    }
    

    Output:

    Array
    (
        [0] => Array
            (
                [price] => 1,200
                [deposittime] => QAR
                [category] => Tablets, West Bay
                [name] => iPad Air 64gb Silver with Leather cover
                [addedby] => rocknrolla
            )
    
        [1] => Array
            (
                [price] => 2,500
                [deposittime] => QAR
                [category] => Mobile phones, Fereej Al Ameer / Muraikh
                [name] => iPhone 6 Plus 64gb
                [addedby] => nabbool
            )
    
        [2] => Array
            (
                [price] => 2,300
                [deposittime] => QAR
                [category] => Mobile phones, Al Sadd
                [name] => lady use 6 plus gold 64 gb
                [addedby] => nijumok
            )
    
        [3] => Array
            (
                [price] => 2,050
                [deposittime] => QAR
                [category] => Mobile phones, Old Airport
                [name] => LG v10 blue for sale
                [addedby] => ramsah92
            )
    
        [4] => Array
            (
                [price] => 1,750
                [deposittime] => QAR
                [category] => Mobile phones, Industrial Area
                [name] => Neat and cleaned iPhone 6 16gb
                [addedby] => ali murtza
            )
    
        [5] => Array
            (
                [price] => 1,350
                [deposittime] => QAR
                [category] => Mobile phones, Ain Khaled
                [name] => Brand new honour 7 sell or sawp...4g with 1 year warenty
                [addedby] => makbool_khan
            )
    
        [6] => Array
            (
                [price] => 250
                [deposittime] => QAR
                [category] => Mobile phones, Al Sadd
                [name] => NOTE 3 ACCESSORIES
                [addedby] => MRS70
            )
    
        [7] => Array
            (
                [price] => 0
                [deposittime] => QAR
                [category] => Tablets, West Bay
                [name] => Hi, I'm looking for a Sony Xperia Z4 Tablet
                [addedby] => carl_albrecht
            )
    
        [8] => Array
            (
                [price] => 50
                [deposittime] => QAR
                [category] => Mobile phones, Doha
                [name] => SN0009 -Luxury Ultra-thin Shockproof Armor Back Case Cover for Apple iPhone 6S 
                [addedby] => Qesale
            )
    
        [9] => Array
            (
                [price] => 75
                [deposittime] => QAR
                [category] => Mobile phones, Doha
                [name] => SN0003 - Dual Fast Adaptive USB Car Charger Adapter + Lightning Cable for iPhone Samsung
                [addedby] => Qesale
            )
    
        [10] => Array
            (
                [price] => 2,000
                [deposittime] => QAR
                [category] => Mobile phones, Al Gharrafa
                [name] => iPhone 6 128 like new
                [addedby] => professional76
            )
    
        [11] => Array
            (
                [price] => 75
                [deposittime] => QAR
                [category] => Mobile phones, Other
                [name] => Virtual Reality Cardboard [NEW]
                [addedby] => 1StopGulf
            )
    
    )