Search code examples
pythonurllib2

loop urllib to download csv data from web

I am trying to use a for loop to download all the data at this web address from 2013 to 2009.

The web adress:

http://data.wa.aemo.com.au/datafiles/balancing-summary/balancing-summary-2013.csv
.
.
.
http://data.wa.aemo.com.au/datafiles/balancing-summary/balancing-summary-2019.csv

My code is this:

year = 2006
max_year = 2019
host = "http://data.wa.aemo.com.au/datafiles/balancing-summary/balancing-summary-"
ending = ".csv"

while year < max_year:
    url = host + str(year)
    print(url)
    urllib.urlretrieve(url, url.lstrip(host))
    print("Done" + url)

However it is not downloading the data, but I don't get an error when the script runs?

Any help would be appreciated, thanks.

Solution

  • You forgot to increment the year in your while loop. you also forgot "ending" in url variable. This seems to work for me.

    year = 2006
max_year = 2019
host = "http://data.wa.aemo.com.au/datafiles/balancing-summary/balancing-summary-"
ending = ".csv"

while year < max_year:
    url = host + str(year)+ending
    print(url)
    urllib.urlretrieve(url,url.lstrip(host))
    print("Done" + url)
    year +=1