Boot up scrapyd failed with default configuration: sqlite3.OperationalError: unable to open database file

I just installed scrapyd on Ubuntu (with apt-get tool). However, without doing any change to the configuration, when I launched "scrapyd" I got the following error:

(! 397)-> scrapyd
Unhandled Error
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/application/app.py", line 642, in run
    runApp(config)
  File "/usr/lib/python2.7/dist-packages/twisted/scripts/twistd.py", line 23, in runApp
    _SomeApplicationRunner(config).run()
  File "/usr/lib/python2.7/dist-packages/twisted/application/app.py", line 376, in run
    self.application = self.createOrGetApplication()
  File "/usr/lib/python2.7/dist-packages/twisted/application/app.py", line 441, in createOrGetApplication
    application = getApplication(self.config, passphrase)
--- <exception caught here> ---
  File "/usr/lib/python2.7/dist-packages/twisted/application/app.py", line 452, in getApplication
    application = service.loadApplication(filename, style, passphrase)
  File "/usr/lib/python2.7/dist-packages/twisted/application/service.py", line 405, in loadApplication
    application = sob.loadValueFromFile(filename, 'application', passphrase)
  File "/usr/lib/python2.7/dist-packages/twisted/persisted/sob.py", line 210, in loadValueFromFile
    exec fileObj in d, d
  File "/usr/lib/pymodules/python2.7/scrapyd/txapp.py", line 3, in <module>
    application = get_application()
  File "/usr/lib/pymodules/python2.7/scrapyd/__init__.py", line 14, in get_application
    return appfunc(config)
  File "/usr/lib/pymodules/python2.7/scrapyd/app.py", line 22, in application
    poller = QueuePoller(config)
  File "/usr/lib/pymodules/python2.7/scrapyd/poller.py", line 13, in __init__
    self.update_projects()
  File "/usr/lib/pymodules/python2.7/scrapyd/poller.py", line 30, in update_projects
    self.queues = get_spider_queues(self.config)
  File "/usr/lib/pymodules/python2.7/scrapyd/utils.py", line 18, in get_spider_queues
    d[project] = SqliteSpiderQueue(dbpath)
  File "/usr/lib/pymodules/python2.7/scrapyd/spiderqueue.py", line 12, in __init__
    self.q = JsonSqlitePriorityQueue(database, table)
  File "/usr/lib/pymodules/python2.7/scrapyd/sqlite.py", line 95, in __init__
    self.conn = sqlite3.connect(self.database, check_same_thread=False)
sqlite3.OperationalError: unable to open database file

Failed to load application: unable to open database file

I am pretty new to Python, I was trying to print the SQlite filename that it tried to open but this file is owned by scrapy user and I try to stay away messing with it. Also, here the content of my configuration:

(! 403)-> cat /etc/scrapyd/conf.d/000-default
[scrapyd]
http_port  = 6800
debug      = off
#max_proc  = 1
eggs_dir   = /var/lib/scrapyd/eggs
dbs_dir    = /var/lib/scrapyd/dbs
items_dir  = /var/lib/scrapyd/items
logs_dir   = /var/log/scrapyd

Does anyone have clue about this? Thanks.

Solution

I suspect the Scrapyd configuration is referencing files or folders that your system user doesn't have the appropriate permissions to use.

There are at least three potential solutions to this problem:

Change the dbs_dir in the configuration file (/etc/scrapyd/scrapyd.conf) to a location your system user can access.
Change the permissions on /var/lib/scrapyd/dbs to allow users or groups access.
Run scrapyd as another user, as a superuser, or as root.