Instagram-API ::: cURL error 18: transfer closed with 3789 bytes remaining to read

Question

I get this error in the middle of data mining from instagram (basically ~8000 images and comments were retrieved correctly and suddenly I receive the following error):

 cURL error 18: transfer closed with 3789 bytes remaining to read (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)

The only part I have used curl in my code is:

function url_exists($url) {
    if (!$fp = curl_init($url)) return false;
    return true;
}

and the url is used here:

        $feed_img_url = $feed[$idx]->getImageVersions2()->candidates[0]->getUrl()."\n";
        if (url_exists($feed_img_url)==true) {
            $img = "results/".$feed_id_str."/".$feed_id_str.".jpeg";
            file_put_contents($img, file_get_contents($feed_img_url));
        }

It doesn't tell which line is producing the error but I guess this exception is coming from one of the above as I haven't used the url anywhere else. This part $feed[$idx]->getImageVersions2()->candidates[0]->getUrl()."\n"; is from Instagram PHP API as in https://github.com/mgp25/Instagram-API

Please suggest fixes to this problem.

Further information: This happens when retrieving data from https://www.instagram.com/gegengrader/ while it doesn't have many posts, posts have lots of likes and only 29 of the posts(images) were retrieved. That said, I am not confident if this is an API rate-limit problem or not. If it is, let me know how to get it fixed.

Answer 1

So I realized when I am browsing this instagram account manually, not everything loads anyways and it takes a long time to load it. I used these and now at least I retrieve 70 of 130ish feeds:

function url_exists($url) {
    if (!$cURL = curl_init($url)) {
        return false;
    }

    curl_setopt($cURL, CURLOPT_HTTPHEADER, array('Expect:'));
    return true;
}

and

catch (Exception $e) {
    echo $e->getMessage();
    if (strpos($e->getMessage(), 'cURL error 18: transfer closed') !== false) {
        continue;
    }

}

Possibly not the best solution but it serves my needs. Please feel free to add your answers.