Search code examples
pythoncurldownload-speed

Why is the file downloading work slowly when I use my callback progress with curl?

I have this simple python script, that downloads file using curl. Also it calculates bandwidth. Here is the whole script:

import pycurl
import time

next_time = 0

# Callback function invoked when download/upload has progress
def progress(download_t, download_d, upload_t, upload_d):
    global next_time
    if time.time() >= next_time:
        print("Total to download", download_t)
        print("Total downloaded", download_d)
        print("Total to upload", upload_t)
        print("Total uploaded", upload_d)
        download_speed = (download_d / 1000000 * 8) / 10
        print(" download_speed", download_speed)
        next_time = time.time() + 10


c = pycurl.Curl()
c.setopt(c.URL, "https://mysample_domain.com/speed-test/test/1G.dat")
c.setopt(c.NOPROGRESS, False)
c.setopt(c.XFERINFOFUNCTION, progress)
start_time = time.time()
next_time = time.time() + 10
c.perform()
print('after download, duration = ', time.time() - start_time)

The problem is, when I use my callback progress, file downloads in 459 seconds. When I removed this two lines, file was downloaded in 210 seconds. Why do my calculations in method progress freeze download speed ? 

c.setopt(c.NOPROGRESS, False)
c.setopt(c.XFERINFOFUNCTION, progress)
Solution

  • I've profiled your code, and came to the conclusion that the progress function is not causing any considerable slowdown. In fact, your code only spends 0.2% of the total run-time in the progress function (with some 40k calls during a 20s download).

    I also ran the download multiple times, and noticed that sometimes the download without progress, and sometimes a download with progress took significantly longer than usual (about 2-3 times as long). This might be due to network issues/bottlenecks, or some issue with pycurl, but it's definitely not caused by your progress function.