Search code examples
scrapycommand-line-argumentszyte

How can I add a new spider arg to my own template in Scrapy/Zyte

I am working on a paid proxy spider template and would like the ability to pass in a new argument on the command line for a Scrapy crawler. How can I do that?

Solution

  • This is achievable by using kwargs in your spider's __init__-Method:

    import scrapy


class YourSpider(scrapy.Spider):
    name = your_spider

    def __init__(self, *args, **kwargs):
        super(YourSpider, self).__init__(*args, **kwargs)
        self.your_arg = kwargs.get("your_cmd_arg", 42)

    Now it would be possible to call the spider as follows:
    scrapy crawl your_spider -a your_cmd_arg=foo

    For more information on the topic, feel free to check this page in the Scrapy documentation.