How to get historical weather data (Temperature) on hourly basis for Chicago, IL

Question

I need the historical weather data (Temperature) on hourly basis for Chicago, IL (Zip code 60603)

Basically i need it for the month of June and July 2017 either hourly or in 15 mins interval.

I have searched on NOAA, Weather underground etc. But haven't found anything relevant to my use case. Tried my hands on scraping using R and Python, but no luck.

Here is a snippet for the same

R :

library(httr)
library(XML)
url <- "http://graphical.weather.gov/xml/sample_products/browser_interface/ndfdXMLclient.php"
response <- GET(url,query=list(zipCodeList="10001",
                           product="time-series",
                           begin=format(Sys.Date(),"%Y-%m-%d"),
                           Unit="e",
                           temp="temp",rh="rh",wspd="wspd"))
doc   <- content(response,type="text/xml", encoding = "UTF-8")   # XML document with the data
# extract the date-times
dates <- doc["//time-layout/start-valid-time"]
dates <- as.POSIXct(xmlSApply(dates,xmlValue),format="%Y-%m-%dT%H:%M:%S")
# extract the actual data
data   <- doc["//parameters/*"]
data   <- sapply(data,function(d)removeChildren(d,kids=list("name")))
result <- do.call(data.frame,lapply(data,function(d)xmlSApply(d,xmlValue)))
colnames(result) <- sapply(data,xmlName)
# combine into a data frame
result <- data.frame(dates,result)
head(result)

Error :

Error in UseMethod("xmlSApply") : 
no applicable method for 'xmlSApply' applied to an object of class "list"

Python :

from pydap.client import open_url

# setup the connection
url = 'http://nomads.ncdc.noaa.gov/dods/NCEP_NARR_DAILY/197901/197901/narr-
a_221_197901dd_hh00_000'
modelconn = open_url(url)
tmp2m = modelconn['tmp2m']

# grab the data
lat_index = 200    # you could tie this to tmp2m.lat[:]
lon_index = 200    # you could tie this to tmp2m.lon[:]
print(tmp2m.array[:,lat_index,lon_index] )

Error :

 HTTPError: 503 Service Temporarily Unavailable

Any other solution is appreciated either in R or Python or any related online dataset link

Answer 1

There is an R package rwunderground, but I've not had much success getting what I want out of it. In all honesty, I'm not sure if that's the package, or if it is me.

Eventually, I broke down and wrote a quick diddy to get the daily weather history for personal weather stations. You'll need to sign up for a Weather Underground API token (I'll leave that to you). Then you can use the following:

library(rjson)

api_key <- "your_key_here"
date <- seq(as.Date("2017-06-01"), as.Date("2017-07-31"), by = 1)
pws <- "KILCHICA403"

Weather <- vector("list", length = length(date))

for(i in seq_along(Weather)){
  url <- paste0("http://api.wunderground.com/api/", api_key,
                "/history_", format(date[i], format = "%Y%m%d"), "/q/pws:",
                pws, ".json")
  result <- rjson::fromJSON(paste0(readLines(url), collapse = " "))
  Weather[[i]] <- do.call("rbind", lapply(result[[2]][[3]], as.data.frame, 
                                          stringsAsFactors = FALSE))
  Sys.sleep(6)
}

Weather <- do.call("rbind", Weather)

There's a call to Sys.sleep, which causes the loop to wait 6 seconds before going to the next iteration. This is done because the free API only allows ten calls per minute (up to 500 per day).

Also, some days may not have data. Remember that this connects to a personal weather station. There could be any number of reasons that it stopped uploading data, including internet outages, power outages, or the owner turned off the link to Weather Underground. If you can't get the data off of one station, try another nearby and fill in the gaps.

To get a weather station code, go to weatherunderground.com. Enter your desired zip code into the search bar

Click on the "Change" link

You can see the station code of the current station, and options for other stations nearby.