Search code examples
node.jsexpress

Except from memory and CPU leaks, what will be reasons for Node.js server might go went down?

I have a Node.js (Express.js) server for my React.js website as BFF. I use Node.js for SSR, proxying some request and cache some pages in Redis. In last time I found that my server time to time went down. I suggest an uptime is about 2 days. After restart, all ok, then response time growth from hour to hour. I have resource monitoring at this server, and I see that server don't have problems with RAM or CPU. It used about 30% of RAM and 20% of CPU.

I regret to say it's a big production site and I can't make minimal reproducible example, cause i don't know where is reason of these error :(

Except are memory and CPU leaks, what will be reasons for Node.js server might go went down?

I need at least direction to search.

UPDATE1:

"went down" - its when kubernetes kills container due 3 failed life checks (GET request to a root / of website)

My site don't use any BD connection but call lots of 3rd party API's. About 6 API requests due one GET/ request from browser

UPDATE2:

Thx. To your answers, guys. To understand what happend inside my GET/ request, i'm add open-telemetry into my server. In longtime and timeout GET/ requests i saw long API requests with very big tcp.connect and tls.connect.

enter image description here

I think it happens due lack of connections or something about that. I think Mostafa Nazari is right. I create patch and apply them within the next couple of days, and then will say if problem gone

I solve problem.

It really was lack of connections. I add reusing node-fetch connection due keepAlive and a lot of cache for saving connections. And its works.

Thanks for all your answers. They all right, but most helpful thing was added open-telemetry to my server to understand what exactly happens inside request.

For other people with these problems, I'm strongly recommended as first step, add telemetry to your project.

https://opentelemetry.io/

PS: i can't mark two replies as answer. Joe have most detailed and Mostafa Nazari most relevant to my problem. They both may be "best answers".

Tnx for help, guys.

Solution

  • Gradual growth of response time suggest some kind of leak. If CPU and memory consumption is excluded, another potentially limiting resources include:

    1. File descriptors - when your server forgets to close files. Monitor for number of files in /proc//fd/* to confirm this. See what those files are, find which code misbehaves.

    2. Directory listing - even temporary directory holding a lot of files will take some time to scan, and if your application is not removing some temporary files and lists them - you will be in trouble quickly.

    3. Zombie processes - just monitor total number of processes on the server.

    4. Firewall rules (some docker network magic may in theory cause this on host system) - monitor length of output of "iptables -L" or "iptables-save" or equivalent on modern kernels. Rare condition.

    5. Memory fragmentation - this may happen in languages with garbage collection, but often leaves traces with something like "Can not allocate memory" in logs. Rare condition, hard to fix. Export some health metrics and make your k8s restart your pod preemptively.

    6. Application bugs/implementation problems. This really depends on internal logic - what is going on inside the app. There may be some data structure that gets filled in with data as time goes by in some tricky way, becoming O(N) instead of O(1). Really hard to trace down, unless you have managed to reproduce the condition in lab/test environment.

    7. API calls from frontend shift to shorter, but more CPU-hungry ones. Monitor distribution of API call types over time.