Search code examples
pythonmiddlewarescrapyhttp-cachingscrapyd

HTTPCACHE not working when deploying scrapy project to scrapyd

When I try to use HTTPCACHE with scrapyd I get the following error:

[scrapy] WARNING: Disabled Httpcache Middlware: unable to find scrapy.cfg file to infer project data dir

Solution

  • The problem is that http caching defaults to using the relative path httpcache.

    This works when you run the spider normally from the command line, but not when running it as a service.

    The solution is to set the HTTPCACHE_DIR[docs] setting to an absolute path.