Search code examples
pythonpython-3.ximportscrapyscrapy-item

Relative error while importing items in Scrapy


I'm learning Python and how to use Scrapy. I've been scraping "http://quotes.toscrape.com/" and everything worked well until I tried to organized data using items. It seems the error is

from ..items import QuotetutorialItem

ImportError: attempted relative import with no known parent package"

I have not been able to find the proper way to import items so your help would be highly appreciated!

Here is the code of my spiders:

import scrapy
from ..items import QuotetutorialItem

class QuoteSpider(scrapy.Spider):
    name = "quotes"                                                    
    start_urls = ["http://quotes.toscrape.com/"]

    def parse(self, response):

         items = QuotetutorialItem()                                  
         all_div_quotes = response.css("div.quote")                   

         for quotes in all_div_quotes:                                

            title = quotes.css("span.text::text").extract()
            author = quotes.css(".author::text").extract()
            tag = quotes.css(".tag::text").extract()

            items["title"] = title
            items["author"] = author
            items["tag"] = tag

            yield items

The code of items.py:

import scrapy


class QuotetutorialItem(scrapy.Item):
    # define the fields for your item here like:
    title = scrapy.Field()
    author = scrapy.Field()
    tag = scrapy.Field()

And here the structure:

enter image description here


Solution

  • Why I generally advise is to append your project's path to PYTHONPATH:

    export PYTHONPATH="${PYTHONPATH}:/path/to/quotetutorial/"
    

    and inside your quotes_spider.py use:

    from quotetutorial.items import QuotetutorialItem
    

    Now if you want to import package in items.py from say quotes_spider.py, the following should do the trick:

    from quotetutorial.spiders.quotes_spider import QuotetutorialItem
    

    Alternatively, the following will also work:

    export PYTHONPATH="${PYTHONPATH}:/path/to/quotetutorial/quotetutorial/"
    

    and then:

    from items import QuotetutorialItem
    

    or

    from spiders.quotes_spider import QuotetutorialItem