Search code examples
javamultithreadingdroolsdrools-guvnor

JVM running out of connections resulting into high CPU utilization and OutOfMemoryException

We have a 64 bit linux machine and we make multiple HTTP connections to other services and Drools Guvnor website(Rule engine if you don't know) is one of them. In drools, we create knowledge base per rule being fired and creation of knowledge base makes a HTTP connection to Guvnor website.

All other threads are blocked and CPU utilization goes up to ~100% resulting into OOM. We can make changes to compile the rules after 15-20 mins. but I want to be sure of the problem if someone has already faced it.

I checked for "cat /proc/sys/kernel/threads-max" and it shows 27000 threads, Can it be a reason?

I have a couple of question:

  1. When do we know that we are running over capacity?
  2. How many threads can be spawned internally (any rough estimate or formula relating diff parameters will work)?
  3. Has anyone else seen similar issues with Drools? Concurrent access to Guvnor website is basically causing the issue.

Thanks,

Solution

  • I checked for "cat /proc/sys/kernel/threads-max" and it shows 27000 threads, Can it be a reason?

    That number does look large but we dont know if a majority of those threads belong to you java app. Create a java thread dump to confirm this. Your thread dump will also show the CPU time taken by each thread.

    When do we know that we are running over capacity?

    You have 100% CPU and an OOM error. You are over capacity :) Jokes aside, you should monitor your HTTP connection queue to determine what you are doing wrong. Your post says nothing about how you are handling the HTTP connections (presumably through some sort of pooling mechanism backed by a queue ?). I've seen containers and programs queue requests infinitely causing them to crash with a big bang. Plot the following graphs to isolate your problem

    1. The number of blocking threads over time
    2. Time taken for each thread
    3. Number of threads per thread pool and how they increase / decrease with time (pool size)

    How many threads can be spawned internally (any rough estimate or formula relating diff parameters will work)?

    Only a load test can answer this question. Load your server and determine the number of concurrent users it can support at 60-70% capacity. Note the number of threads spawned internally at this point. That is your peak capacity (allowing room for unexpected traffic)

    Has anyone else seen similar issues with Drools? Concurrent access to Guvnor website is basically causing the issue

    I cant help there since I've not accessed drools this way. Sorry.