Search code examples
phpcurlrssrss-reader

Tricky rss to read with curl


Does any body know hot to make this feed http://maxhire.net/cp/?EA5E6F361D4364703D044F72 to be read with curl? i clearly miss some curl conf, but i'm new to this, usually do JS

function url_get_contents ($Url) {

if (!function_exists('curl_init')){ 
    die('CURL is not installed!');
}
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $Url);
curl_setopt($ch, CURLOPT_POST, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$output = curl_exec($ch);
curl_close($ch);
return $output;
}

And calling the script as below,

echo url_get_contents('http://maxhire.net/cp/?EA5E6F361D4364703D044F72');

does not work for THIS FEED and works with any other, say http: / / x ml.corriereobjects.it/rss/homepage.xml


Solution

  • This website seems to expect a cookie named AspxAutoDetectCookieSupport, if it didn't find it it will redirect you to some cookie detection page, and it will stuck in a loop:

    > curl -I -L http://maxhire.net/cp/?EA5E6F361D4364703D044F72
    HTTP/1.1 302 Found
    Date: Fri, 23 Aug 2013 23:10:55 GMT
    Server: Microsoft-IIS/6.0
    P3P: CP="CAO PSA OUR"
    X-Powered-By: ASP.NET
    X-AspNet-Version: 4.0.30319
    Location: /cp/?EA5E6F361D4364703D044F72&AspxAutoDetectCookieSupport=1
    Cache-Control: private
    Content-Type: text/html; charset=utf-8
    Content-Length: 180
    Connection: Keep-Alive
    Set-Cookie: AspxAutoDetectCookieSupport=1; path=/
    
    HTTP/1.1 302 Found
    Date: Fri, 23 Aug 2013 23:10:56 GMT
    Server: Microsoft-IIS/6.0
    P3P: CP="CAO PSA OUR"
    X-Powered-By: ASP.NET
    X-AspNet-Version: 4.0.30319
    Location: /cp/?EA5E6F361D4364703D044F72&AspxAutoDetectCookieSupport=1
    &AspxAutoDetectCookieSupport=1
    Cache-Control: private
    Content-Type: text/html; charset=utf-8
    Content-Length: 214
    Connection: Keep-Alive
    Set-Cookie: AspxAutoDetectCookieSupport=1; path=/
    
    HTTP/1.1 302 Found
    Date: Fri, 23 Aug 2013 23:10:57 GMT
    Server: Microsoft-IIS/6.0
    P3P: CP="CAO PSA OUR"
    X-Powered-By: ASP.NET
    X-AspNet-Version: 4.0.30319
    Location: /cp/?EA5E6F361D4364703D044F72&AspxAutoDetectCookieSupport=1
    &AspxAutoDetectCookieSupport=1&AspxAutoDetectCookieSupport=1
    Cache-Control: private
    Content-Type: text/html; charset=utf-8
    Content-Length: 248
    Connection: Keep-Alive
    Set-Cookie: AspxAutoDetectCookieSupport=1; path=/
    
    
    ^C
    

    So you need to set this cookie: AspxAutoDetectCookieSupport=1:

    curl_setopt($ch, CURLOPT_COOKIE, 'AspxAutoDetectCookieSupport=1');
    

    That solved the first problem, another problem came up, if you didn't set a value for the user-agent it will send you this page:

    <html xmlns:atom="http://www.w3.org/2005/Atom">
    <head><meta http-equiv="Content-Type" content="text/xml; charset=iso-8859-1" /><
    title>
            Untitled Page
    </title><link href="App_Themes/Default/Common.css" type="text/css" rel="styleshe
    et" /><link href="App_Themes/Default/Container.css" type="text/css" rel="stylesh
    eet" /><link href="App_Themes/Default/Content.css" type="text/css" rel="styleshe
    et" /><link href="App_Themes/Default/Login.css" type="text/css" rel="stylesheet"
     /></head>
    <body>
        <form name="form1" method="post" action="rssCurrentJobs.aspx?site=5E6F361D43
    64703D044F72" id="form1">
    <input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="/wEPDwUKMTc2MTg4
    NDc4NmRk" />
    
        <div>
    
        </div>
        </form>
    </body>
    </html>
    

    So add a user-agent value:

    curl_setopt($ch, CURLOPT_USERAGENT, "SomeUserAgent");
    

    Full code:

    function url_get_contents ($Url) {
    
        if (!function_exists('curl_init')){ 
            die('CURL is not installed!');
        }
    
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL, $Url);
        curl_setopt($ch, CURLOPT_POST, 0);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
        curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
        curl_setopt($ch, CURLOPT_USERAGENT, "SomeUserAgent");
        curl_setopt($ch, CURLOPT_COOKIE, 'AspxAutoDetectCookieSupport=1');
        $output = curl_exec($ch);
        curl_close($ch);
        return $output;
    }