Below is a snippet that searches HackerNews api for the keyword 'java' by making multiple requests to retrieve the first N pages of data
Ideally it should retrieve the first 3 pages of the data set:
https://hn.algolia.com/api/v1/search_by_date?query=java&page=0
https://hn.algolia.com/api/v1/search_by_date?query=java&page=1
https://hn.algolia.com/api/v1/search_by_date?query=java&page=2
Here is code, which is also available as a CodePen
// Fetch all HackerNews posts with the search string 'java'
const url = 'https://hn.algolia.com/api/v1/search_by_date';
const config = {
headers: { 'Content-Type': 'application/json', 'Accept': 'application/json' },
params: { query: 'java', page: null }
};
const numPagesToFetch = 3;
// Build a series of requests, one for each page
let requests = [];
for (var i = 0; i < numPagesToFetch; i++) {
// Duplicate the `config` object for each request
let requestCfg = Object.assign({}, config);
// The only `param` changing for each request is `?page=`
// HN API starts numbering from page 0
requestCfg.params.page = i;
console.log(`Generating request for page ${requestCfg.params.page}`);
requests.push(axios.get(url, requestCfg));
}
// Wait on all requests to complete in parallel, then handle the output
Promise.all(requests).
then((responses) => {
// Should be equal to numPagesToFetch (3)
console.log(`Number of responses: ${responses.length}`);
for(var i = 0; i < responses.length; i++) {
// Print out some info about each response
const firstId = responses[i].data.hits[0].story_id;
const requestUrl = responses[i].request.responseURL;
console.log(`Dataset Response from ${requestUrl} starts with id ${firstId}`);
}
}).
catch(error => {
console.log('There was an error');
})
When I run this, it seems to be retrieving the same page (the last page) multiple times
https://hn.algolia.com/api/v1/search_by_date?query=java&page=2
https://hn.algolia.com/api/v1/search_by_date?query=java&page=2
https://hn.algolia.com/api/v1/search_by_date?query=java&page=2
I feel like it has something to do with how JS or axios handles references to variables. As I build each request and increment the page
, it's impacting all the other requests constructed with that same variable. I even took care to "duplicate" the config
object, but it doesn't help.
Any idea why it's making the same request 3 times?
Thanks!
This is because you are not cloning it deeply, you only clone the root object but the inner objects still point to their original object...you can either use https://www.npmjs.com/package/clone-deep or use this little trick of converting it to a string then parsing as object:
let requestCfg = JSON.parse(JSON.stringify(config));