Search code examples
pythonhtmlurlurllib2

What is the difference between saving web page info with Python program(urllib, request) and save through browser


Span class names and div class information of HTML page is missing when i am trying to open the Web page with the python program. I have tried urllib.request.urlopen() , requests.get(), wget, robobrowser and some other packages in saving the web page with data of class names,but not succeeded.

url = 'https://www.google.com/maps/dir/Navallur,+Tamil+Nadu+600130/Vijayawada,+Andhra+Pradesh/@14.6711659,78.0012123,7z/data=!3m1!4b1!4m13!4m12!1m5!1m1!1s0x3a525a51439fd9f3:0x5fdacd19ed90126c!2m2!1d80.225463!2d12.8447728!1m5!1m1!1s0x3a35eff9482d944b:0x939b7e84ab4a0265!2m2!1d80.6480153!2d16.5061743'

page = urllib.request.urlopen(url).read()
page1 = requests.get(url).content

html = open('some1.html','wb')
html.write(page)
html.close()

Solution

  • Selenium package in Python allows you to do the task Naga :-)

    from selenium import webdriver
    
    ff = webdriver.Firefox()
    ff.get(URL)
    html = ff.page_sources