Search code examples
pythonpython-2.7web-scrapingscrapypipeline

Skip processing in Pipeline If argument met


I'm grabbing product info from a site and checking o see if it's in stock and shippable or not. If it's out of stock o not available to ship then I don't care about adding it to my database. So, I have the item and it's makes its way to the pipeline and once I see that it's out of stock or doesn't ship I just want to quit processing it in the pipeline and dump everything I have on it and go back to the spider to grab the next item to be processed. I hope that makes since. I tried the raise DropItem() but nothing happened. I checked the DB and when that field is empty it should be skipping it right? Here's the code.

Ships = item['Ships']
Stock = item['InStock']
if "shipping not available" in Ships[0].lower():
    raise DropItem()
if len(Stock) is 0:
    raise DropItem ("Test")
if "instock" not in Stock[0].lower():
    raise DropItem()

I've tested the output of the code and it's working correctly as far as the if statement goes. It's just not working when I get to the THEN part. So, to summarize one of the examples in there, When len(Stock) is 0 then I want to stop processing this particular item and move on to getting and processing the next item.


Solution

  • I got it. There was a line of code in there that was messing everything up.

    from __future__ import print_function
    

    When that line of code was in my pipeline, I had nothing in my results of my print. I removed that line and it all came together. Thanks for the suggestions.