Search code examples
pythonweb-scrapingprofilinggoogle-search

Use GoogleSearch on Python with a profiling user?

I'm trying to scrape the web using Google Search in Python, but I would like it to take into account my past searches. Is there a way to login and then search in Google using Python?

This is part of the code I'm using:

from googlesearch import search

for j in search(query, stop=n, lang=lang):
    #request al browser
    req = urllib.request.Request(j, headers={'User-Agent': "Magic Browser"})
    cj = CookieJar()
    try:
        opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
    except:
        pass
Solution

  • Try this code (please notify: googlesearch do not support proxies, but i know how add this feature if you need):

    from googlesearch import search
query = "apple iphone news 2019"
for i in search(\
    query,        # The query you want to run
    tld  = 'com.ua', # The top level domain
    lang = 'en',  # The language
    num  = 10,    # Number of results per page
    start= 0,     # First result to retrieve
    stop = None,  # Last result to retrieve
    pause= 2.0,   # Lapse between HTTP requests
    ):
    my_results_list.append(i)
    print(i)

    Please took closely attension with 'tld' param - thay must be equal to top-level google domain (domain level rises from right to left). For example for my territory Google server DNS eq 'www.google.com.ua' and 'tld'='com.ua'. Refer this list for complete list of Google servers for each territory

    Updated: For support proxies:

    proxy_support = urllib.request.ProxyHandler({'http' : 'http://IP-Address:Port', 
                                             'https': 'http://IP-Address:port'})
opener = urllib.request.build_opener(proxy_support)
urllib.request.install_opener(opener)