I am using Splash to render javascript. But it is sending incorrect URL. To be precise, it sends preceding url. Have a look at this code.
def parse:
splash_args = {'html': 1, 'png': 0}
url = 'http://quotes.toscrape.com/js'
yield Request(url,
self.parse_result,
meta={'splash': {
'endpoint':'render.html',
'args': splash_args,
'splash_url': 'http://localhost:8050'
}
}
)
url = 'https://www.google.com'
yield Request(url,
self.parse_result,
meta={'splash': {
'endpoint':'render.html',
'args': splash_args,
'splash_url': 'http://localhost:8050'
}
}
)
def parse_result(self, response):
print(response.url)
I have used docker container to run Splash. And in docker logs I see this:
2020-08-02 05:34:09.061509 [events] {"active": 1, "status_code": 200, "args": {"headers": {"User-Agent": "Scrapy/2.2.0 (+https://scrapy.org)", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "en"}, "html": 1, "png": 0, "url": "http://quotes.toscrape.com/js", "uid": 140386374564776}, "client_ip": "172.17.0.1", "qsize": 0, "user-agent": "Scrapy/2.2.0 (+https://scrapy.org)", "load": [0.1, 0.08, 0.06], "path": "/render.html", "fds": 22, "method": "POST", "maxrss": 746168, "rendertime": 0.109375, "_id": 140386374564776, "timestamp": 1596346449}
2020-08-02 05:34:09.062780 [-] "172.17.0.1" - - [02/Aug/2020:05:34:08 +0000] "POST /render.html HTTP/1.1" 200 8974 "-" "Scrapy/2.2.0 (+https://scrapy.org)"
2020-08-02 05:34:09.072852 [events] {"active": 0, "status_code": 200, "args": {"headers": {"User-Agent": "Scrapy/2.2.0 (+https://scrapy.org)", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "en"}, "html": 1, "png": 0, "url": "http://quotes.toscrape.com/js", "uid": 140386500587760}, "client_ip": "172.17.0.1", "qsize": 0, "user-agent": "Scrapy/2.2.0 (+https://scrapy.org)", "load": [0.1, 0.08, 0.06], "path": "/render.html", "fds": 22, "method": "POST", "maxrss": 746168, "rendertime": 0.13172173500061035, "_id": 140386500587760, "timestamp": 1596346449}
2020-08-02 05:34:09.073582 [-] "172.17.0.1" - - [02/Aug/2020:05:34:08 +0000] "POST /render.html HTTP/1.1" 200 8974 "-" "Scrapy/2.2.0 (+https://scrapy.org)"
Both requests have same url to 'quotes.toscrape.com' but no requests to 'www.google.com' is seen to be made.
In stdout too, I see no google.com.
2020-08-02 15:34:09 [scrapy.core.engine] DEBUG: Crawled (200) <POST http://localhost:8050/render.html> (referer: None)
2020-08-02 15:34:09 [scrapy.core.engine] DEBUG: Crawled (200) <POST http://localhost:8050/render.html> (referer: None)
http://quotes.toscrape.com/js
http://quotes.toscrape.com/js
2020-08-02 15:34:09 [scrapy.core.engine] INFO: Closing spider (finished)
response.url has printed only quotes.toscrape.com. I'm sure both of these requests are executed though because we are seeing two requests being made. Just the URL is not correct. Please help.
It seems like using url in Request(url) is not enough. I had to add url in meta['splash']['url'] too and that worked.