Search code examples
pythonpython-2.7web-scrapingscrapyscreen-scraping

Can't scrape some links using Scrapy

I got a weird error. I can't scrape link https://www.example.com/2/

But, I can scrape link https://www.example.com/922/

P.S. I am not using the real link since I am not allowed by my job. Sorry.

Solution

  • When I try to debug it using command scrapy view https://www.example.com/2/, it shows the correct HTML I am expecting. When I check the URL via Chrome inspector, it turns out it gives the correct HTML but gives error 500 status instead of 200. I solved it using handle_httpstatus_list = [500] in my spider.