Search code examples
gitpython-3.xscrapyeggscrapyd

Scrapyd-deploy command didn't take file from git submodules

everyone. I have issues.Here is my project structure:

+-- scraper  
|  +-- scraper  
|    +-- classification  
|    |  +-- classifier.py  
|    |  +-- .gitignore  
|    +-- helpers  
|    |  +-- help1.py  
|    +-- spiders  
|    |  +-- spider1.py
.gitignore
.gitmodule
scrapy.cfg

When I run command scrapyd-deploy scraper -p scraper - I haven't deployed dir classification This is git submodule that not deployed yet. What I do wrong?

Solution

  • Your tree looks like it's missing __init__.py files. Without those files scrapyd will be unable to identify the packages.
    All python packages need to contain __init__.py in their root directory. So your tree should look more like:

    +-- scraper  
|  +-- scraper  
|    +-- __init__.py    <---
|    +-- classification  
|    |  +-- __init__.py    <---
|    |  +-- classifier.py  
|    |  +-- .gitignore  
|    +-- helpers  
|    |  +-- __init__.py    <---
|    |  +-- help1.py  
|    +-- spiders  
|    |  +-- __init__.py    <---
|    |  +-- spider1.py
|.gitignore
|.gitmodule
|scrapy.cfg
|setup.py   <---

    And probably have a setup.py file for setup instructions as well.