In Python how can I download a bunch of files quickly? urllib.urlretrieve() is very slow, and I'm not very sure how to go about this.
I have a list of 15-20 files to download, and it takes forever just to download one. Each file is about 2-4 mb.
I have never done this before, and I'm not really sure where I should start. Should I use threading and download a few at a time? Or should I use threading to download pieces of each file, but one file at a time, or should I even be using threading?
urllib.urlretrieve() is very slow
Really? If you've got 15-20 files of 2-4mb each, then I'd just line 'em up and download 'em. The bottle neck is going to be the bandwith for your server and yourself. So IMHO, hardly worth threading or trying anything clever in this case...