Search code examples
jsonscrapyparameter-passingscrapyd

Passing json arguments to a spider in scrapy

I should pass to a spider some parameters taken from a json file. I have read that it is possible through scrapyd using schedule.json but I don't understand how to pass the json file. Someone of you have any experience?

Solution

  • You don't pass the arguments using a JSON file. Scrapyd has a JSON API where you can pass arguments along with it. (e.g. $ curl http://localhost:6800/schedule.json -d project=myproject -d spider=somespider -d myargument="value")

    You can handle the arguments passed through kwargs:

    class MySpider(Spider):

    name = 'somespider'

    def __init__(self, *args, **kwargs):
        super(MySpider, self).__init__(*args, **kwargs)
        self.myargument = kwargs.get('myargument', '')

    See http://scrapyd.readthedocs.org/en/latest/api.html for more info.