Search code examples
scrapyscrapyd

Scrapy server setup

Im trying to setup a scrapyd server on AWS and am trying to access it from my local machine. So far, ive managed to get scrapyd running on the remote machine.

I know its running because when I do start scrapyd i get start: Job is already running: scrapyd.

What do I need to do, to be able to access it from my local machine?

Solution

  • You could use ssh port fowarding:

    ssh scrapyd.host.com -L 6800:localhost:6800

    And then access scrapyd at http://localhost:6800 from your local machine.