Search code examples
pythonclassweb-scrapingbeautifulsoupfindall

How to use find_all method in bs4 on an object without class


import requests
from bs4 import BeautifulSoup

result=requests.get('http://textfiles.com/stories/').text
soup=BeautifulSoup (result, 'lxml')
stories=soup.find_all('tr')
print (stories)

The find method works but find_all doesn't I'm not sure why maybe it is because it doesn't have a class?


Solution

  • correct code is

    import requests
    from bs4 import BeautifulSoup
    
    result=requests.get('http://textfiles.com/stories/')
    soup = BeautifulSoup(result.content, 'html5lib')
    stories=soup.find_all('tr')
    

    you can access each 'tr' by

    stories[0]
    

    0 can be replaced with any number in list You can also use Pandas eg

    import pandas
    import requests
    from bs4 import BeautifulSoup
    result=requests.get('http://textfiles.com/stories/')
    soup = BeautifulSoup(result.content, 'html5lib')
    df=pandas.read_html(soup.prettify())
    print(df)