Search code examples
pythonscrapyscrapyd

scrapyd - error while running spiders simultaneously

I'm trying to run two scrapy spiders simultaneously using scrapyd. I execute

curl http://localhost:6800/schedule.json -d project=myproject -d spider=somespider

result - {"status": "ok", "jobid": "6487ec79947edab326d6db28a2d86511e8247444"} then:

curl http://localhost:6800/schedule.json -d project=myproject -d spider=somespider2

result - 

    python2.7/site-packages/twisted/web/server.py", line 234, in render
            body = resrc.render(self)
        --- <exception caught here>
        File "/usr/local/lib/python2.7/json/decoder.py", line 383, in raw_decode
raise ValueError("No JSON object could be decoded")
            exceptions.ValueError: No JSON object could be decoded

and everytime I run curl for scrapyd - I get the same - exceptions.ValueError: No JSON object could be decoded. The only one way to proceed working with scrapyd - to restart it. How can I run multiple spiders with scrapyd? Why do I get that error?

I use scrapy 1.1, scrapyd 1.1, python 2.7.9

Solution

  • The problem was in sqlite3 installed in my Centos6.5. Decision: need to add

    self.conn.text_factory = bytes

    to SqliteDict and SqlitePriorityQueue classes after connect creation in a scrapyd/sqlite.py b/scrapyd/sqlite.py