Search code examples
pythongetweb-scrapingpython-requestsscraper

request.get doesn't work in python scraper

Hi I am trying to make this basic scraper work, where it should go to a website fill "City" and "area" ,search for restaurants and return the html page.

This is the code i'm using

payload = OrderedDict([('cityId','NewYork'),('area','Centralpark')])

req = requests.get("http://www.somewebsite.com",params=payload)

f = req.content 
soup = BeautifulSoup((f))

And Here is how the Source HTML looks like

enter image description here

When I'm checking the resulting soup variable it doesn't have the search results , instead it contains the data from the first page only,which has the form for entering city and area value (i.e. www.somewebsite.com, what i want is results of www.somewebsite.com?cityId=NewYork&area=centralPark).So Is there anything that i have to pass with that params to explicitly press the search button or is there any other way to make it work.

Solution

  • You need first check whether you can visit the URL by web browser and get the correct result.