Search code examples
pythonhtmlbeautifulsouppython-requestspython-requests-html

When using 'requests.get()', the html is less than the actual html


enter image description here

enter image description here

The first image is the html of the site brought in python, and the second image is the actual html viewed by pressing F12 on the site. I don't know why the two results are different. Other sites display html normally, but I wonder why only that site is not displayed normally.

Here is the Code:

import requests
from bs4 import BeautifulSoup


result = requests.get('https://www.overbuff.com/heroes')

soup = BeautifulSoup(result.text, "html.parser")

print(soup)

Solution

  • Your maybe blocked by the page and should try it with some headers like:

    headers = {"user-agent": "Mozilla/5.0"}    
    r = requests.get(url, headers=headers)
    

    Example

    import requests
    from bs4 import BeautifulSoup
    url = "https://www.overbuff.com/heroes"
    headers = {"user-agent": "Mozilla/5.0"}
    
    r = requests.get(url, headers=headers)
    
    soup = BeautifulSoup(r.text, "html.parser")
    
    print(soup)