Search code examples
pythoncurllibcurlpycurl

pyCurl spikes CPU when I add option MAX_SEND_SPEED_LARGE

My app uses PyCurl to upload multiple files one by one. It works all good with all cURL options and CPU consumption remains below 1%. When I add MAX_SEND_SPEED_LARGE option to throttle bandwidth, CPU consumption goes to 100%.

c.setopt(c.MAX_SEND_SPEED_LARGE, int(rate_limit))

Further investigation reveals, it only happens on OS X 10.9

Solution
    • Are you working with libcurl < 7.31.0?
    • Also: are you using the progress function?

    If yes, I would say you may have hit a bug that has been fixed in libcurl 7.31.0, i.e curl_easy_perform: avoid busy-looping:

    I am using pycurl in a small script to download coursera videos [...] it hangs 1 core of the machine all the time [...] I can confirm that this occurs, and only when rate limiting for upload/download is in place.

    The patch can be seen here.