Search code examples
scrapyscrapyd

Keep scrapyd running

I have scrapy and scrapyd installed on a debian machine. I log in to this server using a ssh-tunnel. I then start scrapyd by going: scrapyd

Scrapyd starts up fine and I then open up another ssh-tunnel to the server and schedule my spider with: curl localhost:6800/schedule.json -d project=myproject -d spider=myspider

The spider runs nicely and everything is fine.

The problem is that scrapyd stops running when I quit the session where I started up scrapyd. This prevents me from using cron to schdedule spiders with scrapyd since scrapyd isn't running when the cronjob is launched.

My simple question is: How do I keep scrapyd running so that it doesn't shut down when I quit the ssh session.

Solution

  • Run it in a screen session:

    $ screen
$ scrapyd

# hit ctrl-a, then d to detach from that screen

$ screen -r # to re-attach to your scrapyd process