Search code examples
scrapyscrapyd

Dynamic Scrapy settings

I have a single Scrapy project with multiple spiders. This project is hosted on a scrapyd instance. I would like to be able to dynamically change settings in the projects settings.py file (such as DOWNLOADER_MIDDLEWARES).

Is it possible to change these settings at the time of sending a request to the scrapyd instance. Note that I don't want to create multiple projects as this will result in duplicating common code across projects.

Thanks

Solution

  • You can pass parameters to scrapyd and change settings using the -d argument

    curl http://localhost:6800/schedule.json -d project=myproject -d spider=somespider -d setting=DOWNLOAD_DELAY=2 -d arg1=val1