Python Requests Google Custom Site Search Without API

Question

I'm trying to create a webscraper which will get links from Google search result page. Everything works fine, but I want to search a specific site only, i.e., instead of test, I want to search for site:example.com test. The following is my current code:

import requests,re
from bs4 import BeautifulSoup
from urllib.parse import urlparse, parse_qs

s_term=input("Enter search term: ").replace(" ","+")
r = requests.get('http://www.google.com/search', params={'q':'"'+s_term+'"','num':"50","tbs":"li:1"})

soup = BeautifulSoup(r.content,"html.parser")

links = []
for item in soup.find_all('h3', attrs={'class' : 'r'}):
    links.append(item.a['href'])

print(links)

I tried using: ...params={'q':'"site%3Aexample.com+'+s_term+'"'... but it returns 0 results.

Answer 1

Change your existing params to the below one:

params={"source":"hp","q":"site:example.com test","oq":"site:example.com test","gs_l":"psy-ab.12...10773.10773.0.22438.3.2.0.0.0.0.135.221.1j1.2.0....0...1.2.64.psy-ab..1.1.135.6..35i39k1.zWoG6dpBC3U"}