Search code examples
python-3.xscrapyscrapy-pipeline

Scrapy: passing instance variables between pipelines


Does passing spider instance variables between pipelines work? Unfortunately I do not have the code but I'll try to explain as short and clear as possible.

Order is the following: Pipeline_1: high priority (@700) Pipeline_2: low priority (@900)

In Pipeline_1 I'm defining a spider instance variable with spider.variable=[] in init method and filling it in close_spider method of the same pipeline.

In Pipeline_2 I'm accessing it in spider_closed method (tried close_spider too) but it is empty.

In pipeline_1 I'm processing some items and I need to pass them all at once in pipeline_2 and this is the only solution I've been thinking about.


Solution

  • So after reading a bit more I found that defining an instance variable in the pipelines is not that great of an idea (also what Alexander mentioned) - I've solved the issue by implementing the pipeline_2 methods into pipeline_1 and do everything in close_spider method.