Search code examples
rgoogle-apiautomationgoogle-custom-search

Client error: automated custom google search with R

I access the Google Custom Search API via googleAuthR using the httr package to automatically search Google for randomly created strings, e.g. numbers. My contingents are 100 searches per second and 10.000 searches a day. I create a list and send my requests, such as:

random.search.string<- function(n = 10000)
    {randomNumbers <- c(1:n)
    for (i in 1:n)
    {randomNumbers[i] <- paste(sample(0:9, 9), collapse = "")}
    return(randomNumbers)}

mydata<-cbind.data.frame(as.factor(random.search.string()))
mydata[ , 2]<- c(1:length(random.search.string()))

Using this dataframe, I create the queries:

cs1<-"https://www.googleapis.com/customsearch/v1?key="
API_Key<-"myAPI-Key"
ENGINE_ID<-"mySecret"
cs2<-"&googlehost=.de&num=1"

query <- vector()
for (i in mydata[, 2]) query[i] <- paste(cs1, API_Key, "&cx=", ENGINE_ID, "&q=", mydata[i, 1], cs2, sep = "")

I receive the output via: 

output<-list()
for (k in query) {output[k] <- content(GET(k))$items[[1]]$link[1]; flush.console(); Sys.sleep(0.5)}

The output contains all links of searches that had a result. However, sometimes the dashboard of the Google APIs indicates client errors (403): dailyLimitExceeded - although the API Manager indicates that I am about a hundred queries under my daily limit. Any suggestions how to fix that?

Thank you very much in advance.

Solution 
  • {
  "error": {
    "errors": [
      {
        "domain": "usageLimits",
        "reason": "dailyLimitExceeded",
        "message": "Daily Limit Exceeded"
      }
    ],
    "code": 403,
    "message": "Daily Limit Exceeded"
  }
}

    If you are getting the daily limit error then this basically means that you have made the max number of requests for that day. The quota will reset at mid night West cost USA time.

    If you go to the Google Developer console under Library find the API you enabled look for the Quotas tab. You can increase the daily quota on some of the Google API. Click the little pencil on the side to increase it

    enter image description here

    Note: Google is not perfect with calculating your quota. You can be plus or minus a few hundred.