Is there a way, using Nightmare or similar, to access the json response?

Question

I have an issue similar to this, where I'm using Nightmare to login to a page and navigate through it. In this page, some requests get json responses to populate the data (seen in Network tab). Is there any way of accessing this json instead of parsing the page itself?

var Nightmare = require('nightmare');
const nightmare = Nightmare({ show: true });
nightmare
    .goto('https://my.url/')
    .type('#user_id', 'myUserId')
    .type('#password', 'passw0rd')
    .click('#button-login')
    .wait(3000)
    .goto('specific-url') // this URL loads a page with some data
    .end()
    .then(console.log)    // this prints stuff like HTTP response: OK
    .catch((error) => {
    console.error('Search failed:', error);
});

I appreciate any help. Thanks

Answer 1

I couldn't test this myself as I haven't used Nightmare or node installed but based from what I've read here, this might do the trick:

var Nightmare = require('nightmare');
const nightmare = Nightmare({ show: true });
nightmare
    .goto('https://my.url/')
    .type('#user_id', 'myUserId')
    .type('#password', 'passw0rd')
    .click('#button-login')
    .wait(3000)
    .goto('specific-url') // this URL loads a page with some data
    .wait(1000)
    .evaluate(() => {
        var jsonUrl = 'needs to contain the address of the JSON backend';
        var filename = './json-result.json';
        var file = fs.createWriteStream(filename);
        var request = http.get(jsonUrl, function (response) {
          response.pipe(file);
        });
      }
    )
    .end()
    .then(console.log)    // this prints stuff like HTTP response: OK
    .catch((error) => {
      console.error('Search failed:', error);
    });

This requires that you have a static URL that is being called that will then return the JSON response. If you need to pass in additional parameters, you might be better off to use an XMLHttpRequest inside the evaluate() block like described here.