Search code examples
pythonpython-3.xpython-requestspython-requests-html

How to get all visible text in a web page (not html source)?

for example, I'd want to get the text show at "www.google.com" just like open it in chrome and press ctrl+a & ctrl+c:

..   
Google PrivacyTermsSettingsAdvertisingBusinessAboutHow Search works

instead of:

<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en"><head><meta charset="UTF-8"><meta content="origin" name="referrer"><meta content="Search the world's information, including webpages, images, videos and more. Google has many special features to help you find exactly what you're looking for." name="description"><meta content="noodp" name="robots"><meta content="/images/branding/googleg/1x/googleg_standard_color_128dp.png" itemprop="image"><meta content="origin" name="referrer"><title>Google</title><script nonce="kYKSVIWLPxNkDhoVCq276A==">(function(){window.google={kEI:'ZqUZXruXDNfT-
...

I have tried the requests_html model like blow:

import requests_html
s = requests_html.HTMLSession()
page = s.get('https://www.google.com')
print(page.html.text)

but it still show the html like blow:

Google
(function(){window.google={kEI:'y6cZXu3LJ8SkwAPWz6KIBA',kEXPI:'31',authuser:0,kGL:'ZZ',kBL:'JGpW'};google.sn='webhp';google.kHL='en';google.jsfs='Ffpdje';})();(function(){google.lc=[];google.li=0;google.getEI=function(a){for(var b;a&&(!a.getAttribute||!(b=a.getAttribute("eid")));)a=a.parentNode;return b||google.kEI};google.getLEI=function(a){for(var b=null;a&&(!a.getAttribute||!(b=a.getAttribute("leid")));)a=a.parentNode;return b};
...

then, how could I get the all text show on the page like press ctrl+a and ctrl+c?

Thanks.

Solution

  • There several ways of doing this, but the one I usually use is:

    from bs4 import BeautifulSoup as bs
import requests_html
s = requests_html.HTMLSession()
page = s.get('https://www.google.com')
soup=bs(page.text,'lxml')
print(soup.get_text())

    Output:

    About Store GmailImagesSign in Remove Report inappropriate predictions PrivacyTermsSettingsSearch settingsAdvanced searchYour data in SearchHistorySearch HelpSend feedbackAdvertisingBusiness How Search works