Search code examples
phpcurlmemory-leakscurl-multi

Fetching data with curl_multi_exec excessive memory usage

Fetching data with curl_multi_exec causes huge memory usage. I tried debugging this and this is where the memory usage starts rising.

do {
    $mrc = curl_multi_exec($mh, $active);
    var_dump("Curl exec ".memory_get_usage());
} while ($mrc == CURLM_CALL_MULTI_PERFORM);

while ($active) {
    var_dump("Check active ".memory_get_usage());
    if (curl_multi_select($mh) != -1) {
        do {
            $mrc = curl_multi_exec($mh, $active);
        } while ($mrc == CURLM_CALL_MULTI_PERFORM);
    }
}

The urls I'm fetching here are ~3mb of raw JSON formatted data each, and there is four of them so it should in theory take ~12mb of memory, right?

string(17) "Curl exec 1697936"
string(20) "Check active 1697936"
string(20) "Check active 1947536"
string(20) "Check active 2373824"
string(20) "Check active 3012800"
string(20) "Check active 3852224"
string(20) "Check active 4966336"
string(20) "Check active 5801920"
string(20) "Check active 6850800"
string(20) "Check active 7882992"

PHP version 5.3.6

Solution

  • memory_get_usage() returns the value in bytes! Here's the docs http://php.net/manual/en/function.memory-get-usage.php In this light, the values don't seem out of the ordinary, 7,882,992 bytes being 7.88 MB.