Yes - another newbie question from a pure amateur here ... I'm in way over my head - but how else could I learn this stuff quickly?
I'm trying to set up a spider for my work. I've got a brand new, self-hosted, Ubuntu Server 18.04 LTS install running on a virtual server. I've got Scrapy (v 1.5.1) up and running - shell working and everything. Docker installed too. Python 3.5.2 installed. Pip installed. Splash (v 3.2) installed and it appears to be working.
I do encounter some problems installing scrapy-splash: scrapy-splash install
And even though my Splash seems to be up and running ... splash running
... there are absolutely no way that my scrapy-spider will talk to it --> failed spider crawl
I've googled, I've stackoverflowed, I've pulled hairs, I've cursed (yeah - believe it or not) - but I'm at a loss ...
Any advise in any direction will be much appreciated! Best regards from Norway.
make sure you have $HOME/.local/bin in your $PATH environment variable when using the --user flag with pip. One way of doing this is appending
export PATH=$HOME/.local/bin:$PATH
to your $HOME/bash_profile file.
