So I have written normal code that fetches the price of the token from coinGecko APIs and other crypto tokens such as BTC, ETH, and XRP from the factory contract of the pancake-swap but it is the trading platform which is why I am fetching the price every 6 seconds but after a while my server memory gets full.
Do I have to do something on the server side to clean the memory after some time, or is there something to do in my code I don't know much about servers, and I am not sure what my server memory gets used so much
If you guys know about trading concepts that require continuous price fetching your help is highly appreciated
// Process each token periodically
const processTokens = async () => {
try {
for (const token of TOKENS) {
const priceData = await liquidityContract.methods
.getPrice([token.address, token.address])
.call();
const _openprice = parseFloat(priceData.price) / 10 ** 8;
if (["ETH", "BTC", "XRP"].includes(token.symbol)) {
// Special handling for ETH, BTC, and XRP
client.ws.ticker(token.symbol + "USDT", (ticker) => {
feed = formatFeedFromTicker(ticker, _openprice);
updatePriceFeed(token, feed, callsendprice, tokenDatas);
});
} else {
// General handling for other tokens
if (
!apiCache[token.coinid] ||
Date.now() - apiCache[token.coinid].timestamp > CACHE_DURATION
) {
const response = await fetch(
`https://pro-api.coingecko.com/api/v3/coins/markets?vs_currency=usd&ids=${token.coinid}&order=market_cap_desc&per_page=1&page=1&sparkline=false&x_cg_pro_api_key=${config.CG_KEY}`
);
if (!response.ok) continue;
const data = await response.json();
if (data.length === 0) continue;
apiCache[token.coinid] = {
data: data[0],
timestamp: Date.now(),
};
}
const feed = formatFeedFromAPI(
_openprice,
apiCache[token.coinid].data
);
updatePriceFeed(token, feed, callsendprice, tokenDatas);
}
}
if (broadcastMessage) {
broadcastMessage({ tokenData: tokenDatas });
}
} catch (error) {
console.error("Error in processTokens: ", error);
} finally {
// Schedule the next run
setTimeout(processTokens, 2000);
}
};
// Start the process
processTokens();
// Transaction checking
if (calltx) {
monitorTransactions(callbacktx);
}
};
My initial code was different its time complexity was O(n^2) so I changed to the above code my memory usage decreased to some extent but I want to see if I can manage memory usage more efficiently
You have quite a few possible reasons such as
CoinGecko API responses are stored in apiCache. If TOKENS is large, this cache can grow unchecked. You need to limit the cache size and periodically clean up old entries.
The WebSocket ticker for ETH, BTC, and XRP is opened but never closed and each call to client.ws.ticker creates a new connection, consuming memory. So only have one connection per token and close it when done.
The setTimeout with a callback to processTokens can create a large number of pending callbacks if the processing of tokens takes longer than expected. This can queue up many setTimeout calls, consuming memory. You need to wait until processTokens function finishes before scheduling the next call
If any promises are rejected without being caught properly, the unhandled rejection might cause memory issues.
Here is a suggested rewrite, note I moved the main part of the URL to the start as well.
const apiROOT = `https://pro-api.coingecko.com/api/v3/coins/markets?vs_currency=usd&order=market_cap_desc&per_page=1&page=1&sparkline=false&x_cg_pro_api_key=${config.CG_KEY}`;
const apiCache = {};
const CACHE_DURATION = 5 * 60 * 1000; // Example cache duration of 5 minutes
const CryptoConversionFactor = 8 ** 10; // No need to calculate this each time
const timeoutMS = 2000; // I would raise this too
let tokenDatas = [];
let wsConnections = {};
// Function to clean up cache
const cleanCache = () => {
const now = Date.now();
for (const coinid in apiCache) {
if (now - apiCache[coinid].timestamp > CACHE_DURATION) {
delete apiCache[coinid];
}
}
};
// Function to manage WebSocket connections
const manageWebSocket = (token, _openprice) => {
if (!wsConnections[token.symbol]) {
wsConnections[token.symbol] = client.ws.ticker(token.symbol + "USDT", (ticker) => {
const feed = formatFeedFromTicker(ticker, _openprice);
updatePriceFeed(token, feed, callsendprice, tokenDatas);
});
}
};
// Function to process tokens periodically
const processTokens = async() => {
try {
for (const token of TOKENS) {
const priceData = await liquidityContract.methods
.getPrice([token.address, token.address])
.call();
const _openprice = parseFloat(priceData.price) / CryptoConversionFactor;
if (["ETH", "BTC", "XRP"].includes(token.symbol)) {
manageWebSocket(token, _openprice);
} else {
if (!apiCache[token.coinid] || Date.now() - apiCache[token.coinid].timestamp > CACHE_DURATION) {
const response = await fetch(`${apiROOT}&ids=${token.coinid}`);
if (!response.ok) continue;
const data = await response.json();
if (data.length === 0) continue;
apiCache[token.coinid] = {
data: data[0],
timestamp: Date.now(),
};
}
const feed = formatFeedFromAPI(_openprice, apiCache[token.coinid].data);
updatePriceFeed(token, feed, callsendprice, tokenDatas);
}
}
if (broadcastMessage) {
broadcastMessage({
tokenData: tokenDatas
});
}
} catch (error) {
console.error("Error in processTokens: ", error);
} finally {
cleanCache();
// Schedule the next run only after the current one finishes
setTimeout(processTokens, timeoutMS);
}
};
// Start the process
processTokens();
// Transaction checking
if (calltx) {
monitorTransactions(callbacktx);
}