Search code examples
javawebspherewebsphere-8

WebSphere Application Server Node Agent Shutting down automatically

Let me explain my environment setup

WAS version : 8.5 OS : AIX

we have 1 DMGR, 6 Node agents and 6 nodes. Recently we faced one problem. node agents are shutting down automatically. Log says "Too many open file". We used LSOF commands to monitor the Node agent process ID.

Some kernal parameter. hardfile limit: 20000 softfile limit: 8126

This is the monitoring output.

Date Server 1 Node Agent PID(6054) Server 2 Node Agent PID(8076) 29/11/2015 1636 1641 30/11/2015 2382 2390 01/12/2015 4527 4534 02/12/2015 5265 5274

Node agent process only keeps on opening the file. it is not closing the file descriptor. IBM Suggested to upgrade to 8.5.5 but its not a feasible option and cannot be done so easily across all the environments.

please provide the solution if you have any.

Thanks in advance.

Solution

  • Increase the nofiles softlimit to at least 10000, as recommended by IBM documentation: http://www-01.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/tprf_tuneaix.html?lang=en

    1. Edit the /etc/security/limits file. Add the following lines to the user account that the WebSphere Application Server process runs on:
    nofiles =  10000
nofiles_hard = 10000
    1. Save the changes.
    2. Restart your AIX system.
    3. To verify the result, type the ulimit -a command on the command line.

    That said, this might just delay your issue, and running on 8.5.0.0 is really not a good idea. Upgrading to the latest version (8.5.5.7) can be done in a single step.