Search code examples
pythonweb-scrapingscrapyscrapy-pipelinescrapy-shell

Invoke scrapy's custom exporter by command line


While trying to resolve my problem (output an ordered Json array by a specific item's field), I've received an answer that suggests me to create a custom exporter for the job.

I'm creating one, but... all the examples that I've find suggest to call it by pipeline, but it seems a bit redundant to me (I've already defined custom behavior on personal exporter... why should I customize with pipeline too?).

What I search is a way to call the custom exporter, once defined, by scrapy shell. For example, to output json I will use:

scrapy crawl myspider -o myjson.json

Does it exists some way to specify my custom exporter for writing file in a similar manner? I've found an experimental feature COMMANDS_MODULE for custom commands, but I'm not sure how to link it to my custom exporter.


Solution

  • You can activate an exporter by using the FEED_EXPORTERS setting.

    FEED_EXPORTERS = {
        'json': 'path.to.SortedJsonItemExporter',
    }