Search code examples
pythonscrapyweb-crawlerscrapy-pipeline

TypeError: expected str, bytes or os.PathLike object, not set - error Scrapy


I am a beginner using Scrapy. I am trying to download image and set pipelines but somethings is error and I can not understand about this.

books.py

class Books2Spider(Spider):
    name = 'books2'
    allowed_domains = ['books.toscrape.com']
    start_urls = ['http://books.toscrape.com']

    def parse(self, response):
        books = response.xpath('//h3/a/@href').extract()
        ...
        pass


    def parse_book(self, response):
        l = ItemLoader(item=BooksCrawlerItem(), response=response)

        title = response.css('h1::text').extract_first()
        price = response.xpath('//*[@class="price_color"]/text()').extract_first()

        image_urls = response.xpath('//img/@src').extract_first()
        image_urls = image_urls.replace('../..', 'http://books.toscrape.com/')

        l.add_value('title', title)
        l.add_value('price', price)
        l.add_value('image_urls', image_urls)

        return l.load_item()

settings.py

ITEM_PIPELINES = {
    'scrapy.pipelines.images.ImagesPipeline': 1
}

IMAGES_STORE = {
    '/home/jaki/Dev/WebScrapingScratch/images'
}

I am crawl this command, scrapy crawl books2. If everything is ok then images will be download. But I am facing the error. The error is,

... if os.path.isabs(uri): # to support win32 paths like: C:\some\dir File "/usr/lib/python3.6/posixpath.py", line 66, in isabs s = os.fspath(s) TypeError: expected str, bytes or os.PathLike object, not set


Solution

  • The IMAGE_STORE settings must be a single path.

    Replace:

    IMAGES_STORE = {
        '/home/jaki/Dev/WebScrapingScratch/images'
    }
    

    with:

    IMAGES_STORE = '/home/jaki/Dev/WebScrapingScratch/images'
    

    {'asdf'} is a set with the string asdf, hence the error message.