Search code examples
pythonwebweb-scrapingbeautifulsoupetherscan

Attribute Error with web scraping using BeautifulSoup

I am trying to crawl data from 'etherscan.io' using BeautifulSoup and Python. Here is the website: https://etherscan.io/txs

page_soups = []
for page in range(1, 51):
  url = 'https://etherscan.io/txs?p=' + str(page)
  print(url)
  req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
  webpage = urlopen(req).read()
  page_soup = soup(webpage, "html.parser").find('tbody').find_all('a')
  page_soups += page_soup

I use a loop to scrape multiple webpages but I can just get data 30 first pages. The 31th one has error as follow

Error

I check that webpage and see that it still has same tag and elements as others. Please help me.

Solution

  • It's because of Cloudflare, I added this inside the loop:

    if page%30 == 0:
    time.sleep(20)

    Apparently, waiting for 20 seconds each 30 pages is enough to not get flagged as a bot.