Search code examples
pythonbeautifulsouphtml-parsing

How to print all text in BeautifulSoup4


I'm building a simple parser with beautifulsoup4. After getting all values I need I'm trying to print them, but getting only one value, not all of them. Website I'm parsing: Click

And here's the code:

import requests
from bs4 import BeautifulSoup as BS

response = requests.get('https://www.dotabuff.com/heroes', headers = {'User-agent': 'your bot 0.1'})
html = BS(response.content, 'html.parser')

a = html.select('div > div' '[class = "name"]')
print(a[0].text)

Solution

  • You can iterate over result of .select() (note: CSS selector [class="name"] can be written as .name):

    import requests
    from bs4 import BeautifulSoup as BS
    
    response = requests.get(
        "https://www.dotabuff.com/heroes", headers={"User-agent": "your bot 0.1"}
    )
    soup = BS(response.content, "html.parser")
    
    for name in soup.select(".name"):
        print(name.text)
    

    Prints:

    Abaddon
    Alchemist
    Ancient Apparition
    Anti-Mage
    
    ...
    
    Witch Doctor
    Wraith King
    Zeus
    

    If you want the result in list form:

    out = [name.text for name in soup.select(".name")]
    print(out)