We are managing around 100 servers (Ubuntu 16.XX - 20.XX) and each server has many processes running on them. These processes are very network heavy and open a lot of external connections through the internet as well as connections between each other. And Linux assigns a file descriptor to each of these connections. And each of these connections spawn a thread too, which as I understand also allocates a file descriptor for the thread. We often sit with around 3000 connections per server, and each one has a thread. So we idle at around 6,000 open file descriptors, which is way above the default 1024 limit of Linux.
We have tried to raise this limit in multiple ways but nothing really sticks. It's very unpredictable and will often just change back to 1024 for no reason. We have added entries into the /etc/security/limits.conf file, and we have even written code that runs a shell command inside these processes that sets the limits every 5 minutes using the ulimit command. But it still defaults back to 1024 every now and then. We have lost clients as a result of this and the financial impact is becoming quite severe!
Here is our /etc/security/limits.conf file
root soft nofile 327680
root hard nofile 655360
root soft nproc 327680
root hard nproc 655360
mysql soft nofile 327680
mysql hard nofile 655360
mysql soft nproc 327680
mysql hard nproc 655360
All these processes run as root (I know this is not exactly the safest way, but for now it is what it is for many reasons).
On startup we have added this command in /etc/rc.local/
ulimit -n 655360
And then we run the same command from inside the processes on an interval of 5 minutes. But we still sometimes get file limit errors.
So the question is: Is there a way to completely disable the limit mechanism in Linux? Even if we deliberately break it in order to disable it. Can we delete some files so that it breaks / cannot run. Maybe set permissions to certain files to stop it from starting etc etc. We are up for any hardcore hacks to stop and break this thing. It's becomming a massive problem now.
To disable limits (never done before) you can search in PAM modules and comment line:
session required pam_limits.so
For set unlimited values you can use command
ulimit -u unlimited
or in /etc/security/limits.conf add lines like:
root soft core unlimited
root hard core unlimited
root soft data unlimited
root hard data unlimited
root soft fsize unlimited
root hard fsize unlimited
...