Combining Scrapy with CasperJS

Question

When I try Splash by using Scrapy-Splash, it doesn't support navigating like a real browser. It only renders the HTML, but when it clicks a button, it doesn't render the next page it navigates to.

From my simple research, web scraping with navigation is only supported by CasperJS. But, is it possible to combine Scrapy and CasperJS

Answer 1

You're confusing browser with testing frameworks, Selenium does exactly that but instead of Firefox, use PhantomJS as a headless browser (which is what CasperJS uses).

Click method

# Assume the button has the ID "submit" :)
driver.find_element_by_id("submit").click()

alexce answer on scrapy with selenium

If you need the content of the requests, use something like BrowserMob as a proxy.