Search code examples
pythonscreen-scrapingscrapy

Determine the scrapy root directory

I'd like to be able to read a file relative to the scrapy root directory, as described in the documentation.

Is there a way of determining this path from inside the spider?

Solution

  • I don't know if scrapy is aware of where it is located on the filesystem, but in my settings.py i add

    PROJECT_ROOT = os.path.abspath(os.path.dirname(__file__))

    then in spider you can say

    from yourscrapyprojecthere import settings

settings.PROJECT_ROOT

    Here I have assumed my project root is wherethe settings.py file is located. If you want to get the directory where scrapy.cfg is you have to go up a directory