Search code examples
pythonjsongoogle-apigoogle-search

Google search with python is sporadically non-accurate and has Type Errors

I am using some code I found here on SO to google search a set of strings and return the "expected" amount of results. Here is that code:

for a in months:
    for b in range(1, daysInMonth[a] + 1):

        #Code

        if not myString:
            googleStats.append(None)
        else:
            try:
            query = urllib.urlencode({'q': myString})
            url = 'http://ajax.googleapis.com/ajax/services/search/web?v=1.0&%s' % query
            search_response = urllib.urlopen(url)
            search_results = search_response.read()
            results = json.loads(search_results)
            data = results['responseData']           
            googleStats.append(data['cursor']['estimatedResultCount'])
        except TypeError:
            googleStats.append(None)
for x in range(0, len(googleStats)):
    if googleStats[x] != None:
        finalGoogleStats.append(googleStats[x])

There are two problems, which may be related. When I return the len(finalGoogleStats), it's different every time. One time it's 37, then it's 12. However, it should be more like 240.

This is TypeError I receive when I take out the try/except:

TypeError: 'NoneType' object has no attribute '__getitem__'

which occurs on line

googleStats.append(data['cursor']['estimatedResultCount'])

So, I just can't figure out why the number of Nones in googleStats changes every time and it's never as low as it should be. If anyone has any ideas, I'd love to hear them, thanks!

UPDATE

When I try to print out data for every think I'm searching, I get a ton of Nones and very, very few actual JSON dictionaries. The dictionaries I do get are spread out across all the searches, I don't see a pattern in what is a None and what isn't. So, the problem looks like it has more to do with GoogleAPI than anything else.

Solution

  • The answer is what I was fearing for a while, but thanks to everyone who tried to help, I upvoted you if anythign was useful.

    So, Google seems to randomly freak out that I'm searching so must stuff. Here's the error they give to me :

    Suspected Terms of Service Abuse ...... responseStatus:403

    So, I guess they put limits on how much I can search with them. What is still strange, though, is that it doesn't happen all the time, I still get sporadic successful searches within the sea of errors. That is still a mystery...