Search code examples
html-helperpycurl

Using pycurl to get next page google results

I have written code to search something on google using pycurl. I would like to be able to use pycurl to do this but all help is greatly appreciated. I am looking for the ability to search a term and then "click the next page button" or "click the indexed numbers at the bottom" using pycurl so I can get more then just the first 10 web results.

Thanks in advance.

Solution

  • There are multiple ways to do this.

    if your base url is http://www.google.co.uk/search?hl=en&q=YOUR_QUERY_STRING

    you can add &num=100 to the end of your url to get first 100 results.

    or if you can add &start=N, where N is a multiple of 10, and resend the webrequest to get the next page

    or you just use XPATH to parse the webpage and get the url of the next page on the search results.