Search code examples
htmlweb-scrapingscrapy

Scrapy shell does not work for https://www.target.com.au/

I have tested that my shell works for other websites, but refuse to work on target.

raise self.value.with_traceback(self.tb)
twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.>]
Solution

  • It looks like the site is ignoring requests based on your user agent string.

    This works:

    scrapy shell target.com.au --nolog --set=USER_AGENT=Mozilla