AttributeError: 'NoneType' object has no attribute 'text'

Question

article_content = []
for url in li:
    response = requests.get(url)
    soup = BeautifulSoup(response.text, "html.parser")
    article_content.append(soup.find("div", {"class": "article-content"}).text)

gives the error:

AttributeError                            Traceback (most recent call last)
Cell In[19], line 5
      3 response = requests.get(url)
      4 soup = BeautifulSoup(response.text, "html.parser")
----> 5 article_content.append(soup.find("div", {"class": "article-content"}).text)

AttributeError: 'NoneType' object has no attribute 'text'

I am working with a list of websites and it throws error like above. My target is to store website scraping and store the article content in a separate text file. Thank you in advance for help!!

Answer 1

Cause of limited details in question here a general answer - There seems to be no element with class article-content, for several reasons:

• no site with article content
• content is rendered dynamically via javascript (use api/xhr or selenium)
• captcha / cloudflare blocked
• ...

Always and first at all check your response / soup if it contains expected HTML

You could use an if-statement to handle the error:

for url in li:
    response = requests.get(url)
    soup = BeautifulSoup(response.text, "html.parser")
    article_content.append(
        soup.find("div", {"class": "article-content"}).text if soup.find("div", {"class": "article-content"}) else 'no element available'
    )