reason of why number of Java threads growing in time

Question

I have a standalone Java console program which is using around 80-170 (160 is a typical number at average load) threads in production. Some code are replaced in the project which is functionally fine but the thread number is growing continually after starting the program. I recognized the problem when it is almost crashed the virtual machine. Then it had 30.000 threads.

I would like to find the reason of this behaviour at the production environment so if there is any useful tool it must be a command line program or should able to run from remote host.

I checked it with 'ps huH p | wc -l' command and VisualVm program.Unfortunately VisualVm remote giving information about the thread numbers only, "Thread" tab is not active

I tried to get useful information from jstack command , but I couldn't find what is the new thread who created etc. from

"Timer-459" #1864 daemon prio=10 os_prio=0 tid=0x0bb6c400 nid=0x3c4f in Object.wait() [0x2a980000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:502)
    at java.util.TimerThread.mainLoop(Timer.java:526)
    - locked <0x4d18bc80> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:505)

or

"pool-133-thread-2" #1877 prio=5 os_prio=0 tid=0x42a2c400 nid=0x3e5e waiting on condition [0x2b118000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x4d18ae80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

I don't get what are they.

Could you help me how could I find out what code creates new threads again and again? Any good tool?

Thank you!

Answer 1

I would say that jstack is a correct tool for such task. You can understand threads nature by thier stack traces and source code. E.g. at java.util.TimerThread.mainLoop(Timer.java:526) means that your code uses Timer class and its thread is waiting for the next schedule. java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) means you use one of ExecutorService implementations and its worker thread is waiting for the next task. Try to analyze how many thread pool threads are alive and why, probably they are the source of your issue.