Search code examples
javascriptpythonsearch-enginehtml

How to download search results from Google custom search?


There are websites with google custom search. But when the search results are out, I am unable to save these results in the native html.

For e.g. I want to search for (ITE) on this website http://www.asiaone.com/A1Home/A1Home.html The result page is:

http://www.asiaone.com/vgn-ext-templating/asiaone/search/A1MultipleSearchResults.jsp?q=%28ITE%29&cx=009076860783986227266%3Atwgr1shu6je

but when i tried to download the source of that result page, the results and the synopsis is not available on the flat HTML.

Is there any library available in python or any other software that can download these synopsis from the search results? I have something like this but the html source doesnt have the synopsis nor the search results:

import urllib
url = "http://www.asiaone.com/vgn-ext-templating/asiaone/search/A1MultipleSearchResults.jsp?q=%28" + "ITE" + "%29&cx=009076860783986227266%3Atwgr1shu6je"
html = urllib.urlopen(url).read().decode('UTF-8')
print html 

Solution

  • I am trying to access: https://www.googleapis.com/customsearch/v1?key=&cx=009076860783986227266:Atwgr1shu6je&q=ITE

    I am entering my own key that I got from https://code.google.com/apis/console/?api=customsearch.

    But, I get invalid value error as my key is not associated with your cx id. You have to put your key here and get the results.

    PS: Dont enter your key here in the comments or the question. It can cost you money (https://code.google.com/apis/console/?api=customsearch:billing)

    You can refer to https://developers.google.com/custom-search/v1/using_rest#WorkingResults. It is clearly mentioned what you need to do to get the results.