I deployed a scrappy/flask project on PythonAnywhere.com (PA) that works on my local machine and almost entirely works on PA. The issue I have is that when I schedule a task to run my spider, no pipelines are enabled. This is obviously an issue as my MongoDB and images pipelines will not update. Any ideas as to why I am having this issue and how to resolve it? I am able to query MongoDB and my site renders as expected, it's just that no pipelines are enabled during a crawl so I am left with an outdated site. Upon closer inspection, it appears that the crawler uses an incorrect bot. See the following outputs; the first from my local machine and the second from PA:
Correct - my local machine:
2019-03-16 15:51:12 [scrapy.utils.log] INFO: Scrapy 1.6.0 started (bot: mycorrectbot)
2019-03-16 15:51:12 [scrapy.utils.log] INFO: Versions: lxml 4.3.2.0, libxml2 2.9.9, cssselect 1.0.3, parsel 1.5.1, w3lib 1.20.0, Twisted 18.9.0, Python 3.7.1 (default, Nov 28 2018, 11:51:47) - [Clang 10.0.0 (clang-1000.11.45.5)], pyOpenSSL 19.0.0 (OpenSSL 1.1.1b 26 Feb 2019), cryptography 2.6.1, Platform Darwin-18.2.0-x86_64-i386-64bit
2019-03-16 15:51:12 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'mycorrectbot', 'DOWNLOAD_DELAY': 0.25, 'NEWSPIDER_MODULE': 'mine.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['mine.spiders'], 'USER_AGENT': 'mine (+https://mine.com)'}
Incorrect - on PA:
2019-03-16 22:57:11 [scrapy.utils.log] INFO: Scrapy 1.6.0 started (bot: scrapybot)
2019-03-16 22:57:11 [scrapy.utils.log] INFO: Versions: lxml 4.3.2.0, libxml2 2.9.9, cssselect 1.0.3, parsel 1.5.1, w3lib 1.20.0, Twisted 18.9.0, Python 3.7.0 (default, Aug 22 2018, 20:50:05) - [GCC 5.4.0 20160609], pyOpenSSL 19.0.0 (OpenSSL 1.1.1b 26 Feb 2019), cryptography 2.6.1, Platform Linux-4.4.0-1075-aws-x86_64-with-debian-stretch-sid
2019-03-16 22:57:11 [scrapy.crawler] INFO: Overridden settings: {}
Structure of project:
├── LICENSE
├── README.md
├── flask_app
│ ├── __init__.py
│ ├── flask_app.py
│ ├── static
│ │ ├── css
│ │ │ └── home.css
│ │ ├── images
│ │ │ └── full
│ │ │ ├── 1.jpg
│ │ │ ├── 2.jpg
│ │ │ ├── 3.jpg
│ │ └── vendor
│ │ ├── bootstrap
│ │ │ ├── css
│ │ │ │ ├── bootstrap.css
│ │ │ │ ├── bootstrap.css.map
│ │ │ │ ├── bootstrap.min.css
│ │ │ │ └── bootstrap.min.css.map
│ │ │ └── js
│ │ │ ├── bootstrap.bundle.js
│ │ │ ├── bootstrap.bundle.js.map
│ │ │ ├── bootstrap.bundle.min.js
│ │ │ ├── bootstrap.bundle.min.js.map
│ │ │ ├── bootstrap.js
│ │ │ ├── bootstrap.js.map
│ │ │ ├── bootstrap.min.js
│ │ │ └── bootstrap.min.js.map
│ │ └── jquery
│ │ ├── jquery.js
│ │ ├── jquery.min.js
│ │ ├── jquery.min.map
│ │ ├── jquery.slim.js
│ │ ├── jquery.slim.min.js
│ │ └── jquery.slim.min.map
│ └── templates
│ └── index.html
├── scrapy_app
│ ├── scrapy_app
│ │ ├── __init__.py
│ │ ├── items.py
│ │ ├── middlewares.py
│ │ ├── pipelines.py
│ │ ├── settings.py
│ │ └── spiders
│ │ ├── __init__.py
│ │ └── mine.py
│ └── scrapy.cfg
└── requirements.txt
It is my understanding that I am unable to call the .py for my scrapy spider in PA Tasks directly and must instead rely on a separate file to make the "scrapy crawl my spider" call I would normally make. However, this doesn't seem to work as expected. I referenced, this doc page but I must be missing something.
Here is the file I created to make the alternate call and point to in PA Task:
import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
from scrapy_app.spiders import mine
class MySpider('my spider'):
# Your spider definition
pass
if __name__ == "__main__":
process = CrawlerProcess(get_project_settings())
process.crawl(mine.MySpider)
process.start() # the script will block here until the crawling is finished
The following is the scheduled task I added on PA (not in project structure - please disregard this fact):
/home/myuserid/.virtualenvs/myvirtualenv/bin/python /home/myuserid/flask_app/flask_app/scheduler.py
Thanks in advance for your assistance!
Wow. I made this far more complicated than I needed. The only variance between my PA and local app was the type of Virtual Environment I was running. I deleted the recommended VE on PA and instead used venv. I then wrote a simple script to activate the VE, run my spider, and then deactivate. Script as follows (scheduler.sh):
#!/bin/bash
cd ~
source myvenv/bin/activate
cd /home/userid/project/scrapy_app/scrapy_app
scrapy crawl my spider
deactivate
All now works as expected despite the terrible guidance on both PA and scrapy. Hope this helps someone else!