IBM-WAS: how to configure webphere to avoid OutOfMemoryError?

Our deployed application runs slowly on websphere server, we find there are many OutOfMemoryError messages in WAS log.

[11/22/17 2:35:27:945 UTC] 00000116 ServletWrappe E
com.ibm.ws.webcontainer.servlet.ServletWrapper service Uncaught
service() exception thrown by servlet action:
java.lang.OutOfMemoryError: Failed to create a thread: retVal
-1073741830, errno 11
    at java.lang.Thread.startImpl(Native Method)
    at java.lang.Thread.start(Thread.java:948)
    at com.sun.jndi.ldap.Connection.<init>(Connection.java:246)
    at com.sun.jndi.ldap.LdapClient.<init>(LdapClient.java:149)
    at com.sun.jndi.ldap.LdapClient.getInstance(LdapClient.java:1626)
    at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2758)
    at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:331)
    at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:204)

Recently, we updated both the application and WAS, but before that, we had never seen such problem. We made our application using IBM Content Navigator (ICN) 2.0.3. as an Image Viewer and deployed it, and also upgraded Websphere to WAS 8.5.5.12, so We cannot define the problem whether caused by the ICN Image Viewer or the new WAS fixpack, but it seems the ICN image viewer is more possible to slow down the system, because we always upgrade WAS with new fixpacks, we haven't seen any of them can cause an OutOfMemoryError.

We checked the server free space during the slow performance, we found there was plenty of memory left. And we also adjusted maximum heap size of websphere to 4096 (which is really high), but it still useless. Currently we don't know where to configure to solve such low performance problem. Please give us some suggestions.

Solution

found the same http://www-01.ibm.com/support/docview.wss?uid=swg21633466

Linux has a maximum allowed process per user limit, we can check this using the "ulimit -u" command. If this value is low(default is 1024), then either make it unlimited or raise it to a high value, say 131072. This section is also highlighted as the "max user processes" section in "ulimit -a" output. Use the following command to set it to unlimited: ulimit -u unlimited
Increase the amount of native memory available by lowering the size of the Java heap using the -Xmx option. The process address space not used by the Java heap is available for the native heap usage. Java heap is set in $TOP/bin/conf/service_mem_settings.ini file for the 6 services and as custom_java_options in $TOP/bin/conf/env_settings.ini file for the back end scripts.
Check space requirements using 'df -k' command.
Lower the number of threads being used.