Scrapyd is service where we can eggify deploy our projects. However I am facing a problem. I have a Project named MyScrapers whose spider classes uses an import statement as follows:
from mylibs.common.my_base_spider import MyBaseSpider
The path to my_base_spider is /home/myprojectset/mylibs/common/my_base_spider
While setting environment variable PYTHONPATH=$HOME/myprojectset/, I am able to run MyScrapers using scrapy command: scrapy crawl MyScrapers.
But when I use scrapyd for deploying MyScrapers by following command: scrapy deploy scrapyd2 -p MyScrapers, I get the following error:
Server response (200): {"status": "error", "message": "ImportError: No module named mylibs.common.my_base_spider"}
Please tell how to make deployed project to use these libs?
I found the answer by adding mylibs to site-packages of python by using setup.py inside mylib folder. That way I could import everything inside mylib in my projects. Actually mylibs were way outside from the location where setup.py of my deploy-able project is present. setup.py looks for packages on same level and inside the folders where it is located.