I am working on building a site around Eve Online's ESI api. I am using NodeJS using axios to make async get calls. I am trying to get the market data for the various in game regions using their api. I've been successful in getting the data and adding it to my database, but I feel that I could be making these calls in a better faster way. Their api returns results in pages with 1000 results per page. For some regions I have to request up to 300+ pages but typical is 100 pages. I've been doing this using a for loop, then just checking if the results array has data, if it doesn't I break the for loop. For some regions this takes seconds 10-20 pages only, for others it cane take a few minutes 50+ pages. I'm trying to speed this up. Right now it takes about 60 minutes to download everything and upload to my DB, with the download taking the bulk of time.
This is my current code to get the data. I first get a list of all in game regions IDs, then use a for loop to loop through each region. Using the current region ID i use that to get all the types of items ID's in that region. This is where my first page problem comes in. Each region can have 1 item being sold or 15000. So I picked 50 as an arbitrary number to loop till, checking each call if it returns with data or not. Then I do the same using the same region ID to get all the orders. This is where my download times are getting impacted. Some regions have little or more orders, with one having 300+ pages which has taken anywhere from 12 minutes on my laptop to 4-5 on an AWS setup.
const regionData = await axios.get(
"https://esi.evetech.net/latest/universe/regions/?datasource=tranquility"
);
//Eg region 10000042
const region = regionData.data;
console.log(region);
for (let r = 0; r < region.length; r++) {
let typeID = [];
let orders = [];
for (let i = 1; i < 50; i++) {
const types = await axios.get(
`https://esi.evetech.net/latest/markets/${region[r]}/types/?datasource=tranquility&page=${i}`
);
if (types.data.length) {
for (let x = 0; x < types.data.length; x++) {
typeID.push(types.data[x]);
}
} else {
break;
}
}
for (let i = 1; i < 500; i++) {
const results = await axios.get(
`https://esi.evetech.net/latest/markets/${region[r]}/orders/?datasource=tranquility&order_type=all&page=${i}`
);
if (results.data.length) {
for (let x = 0; x < results.data.length; x++) {
orders.push(results.data[x]);
}
} else {
break;
}
}
... code to sort through data and upload to database
}
I want to speed up this download process so I don't have to wait for each page to download one at a time.
I'm trying to use async promises to download multiples pages at once, but as each region has different amount of pages I've just picked a max number that suits all, this causes the promises to make 300+ calls at once, which just breaks and causes 500 errors. It has worked great when I limit the number of pages to get to less then 100, and can get that data in a fraction of the time it takes my for loops. I just don't know how to implement a way to cycle through and get all the data without trying to request 300 pages at once.
Here's the current code I'm testing using a promise that maps an array of api links for both types and orders. It works great when it's less then 100 pages, but more I start to get 500 server errors.
let urlsR = [];
let urlsT = [];
const getAllData = async (URLs) => {
return Promise.all(URLs.map(fetchData));
};
function fetchData(URL) {
return axios
.get(URL)
.then(function (response) {
return response.data;
})
.catch(function (error) {
console.log(error)
//return { success: false };
});
}
const regionData = await axios.get(
"https://esi.evetech.net/latest/universe/regions/?datasource=tranquility"
);
const region = regionData.data;
for (let r = 0; r < region.length; r++) {
console.log("Starting API fetch for " + region[r]);
urlsR = [];
urlsT = [];
for (let i = 1; i < 17; i++) {
urlsT.push(
`https://esi.evetech.net/latest/markets/${region[r]}/types/?datasource=tranquility&page=${i}`
);
}
for (let i = 1; i < 316; i++) {
urlsR.push(
`https://esi.evetech.net/latest/markets/${region[r]}/orders/?datasource=tranquility&order_type=all&page=${i}`
);
}
let getTypes = await getAllData(urlsT);
let getMarket = await getAllData(urlsR);
let orders = [];
... Code that sorts through data and uploads to DB
}
My issue here is there is no way to check how many pages per region need to be request so I again picked a top limit and every regions it requests that many pages against the api. The api just returns an empty array if no results found which is good to not break the code with an error. I just don't want to make 100 api calls, to only really get 15 pages and 85 empty arrays, or try to do 300 at once and keep getting errors.
Sorry for the ramble, but Any thoughts or direction is helpful.
Here is a solution:
let getMarket = [];
let maxPageLimit = 300;
let cycle = 50;
for (let i = 1; i < maxPageLimit; i += cycle) {
urlsR = [];
for (let j = i; j < i + cycle; j++) {
urlsR.push(
`https://esi.evetech.net/latest/markets/${region[r]}/orders/?datasource=tranquility&order_type=all&page=${i}`
);
}
getMarket = getMarket.concat(await getAllData(urlsR));
if (len(getMarket[len(getMarket - 1)]) === 0) {
break;
}
}
It basically implements a 50 page cycle. After every 50 pages, it checks if the last page it received was empty. If it was, it breaks the fetch loop. You can set both, the cycle volume and the max number of pages you want to iterate.
This will have far less wasted calls depending upon the cycle volume you use.