Search code examples
node.jsweb-scrapingcheeriorequestjs

Edit redirect uri Request.js after login

I'm using request.js and cheerio.js to scrape a password protected website.

Is it possible to edit the response uri such that I could scrape another page other than the one I'm being redirected to?

Solution

  • You should look at using tough-cookie in combination with request.js.

    This should allow you to create a cookie-jar that stores your session on the password-protected websites and you can continue to request other pages whilst still appearing logged in.

    Usage (from the request.js examples):

    var request = request.defaults({jar: true})