Search code examples
javascriptweb-crawlerhttp-postuser-agentsuperagent

Node package that allows crawling of sites that requires javascript


So i've been searching endlessly for a package that allows crawling of javascript sites. I've been trying selenium but i don't like the whole browser deal.

Is there a solution similar to superagent and request where you do post & get requests, which also has javascript "enabled"?


Solution

  • Take a look at scraperjs dynamicScraper function or phantomjs. They both allow run js without loading whole browser. And here an example for how to wait for some events on page