Thread handling in a java application

Question

I created a service which has in total 11 scheduled jobs running. 3 of them are scheduled by a cron job (2 of them every 15min and the last one every minute). These three tasks are only for monitoring the service (Checking ehCache and RAM used by the JVM). All the other scheduled tasks are annotated with the 'fixedDelay' attribute - so a new thread should only be started if the last one is finished and x time passed, right?

With http://ask.xmodulo.com/number-of-threads-process-linux.html I found out, that I can check the number of threads per process by executing

cat /proc/PID/status

This resulted in the following

Name:   jsvc
Umask:  0022
State:  S (sleeping)
Tgid:   17263
Ngid:   0
Pid:    17263
PPid:   17260
TracerPid:      0
Uid:    99      99      99      99
Gid:    99      99      99      99
FDSize: 8192
Groups: 99 11332 16600 34691 50780 52730 52823 53043 54173
NStgid: 17263
NSpid:  17263
NSpgid: 17260
NSsid:  17260
VmPeak: 35247540 kB
VmSize: 35232620 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:   5679220 kB
VmRSS:   5663344 kB
RssAnon:         5660016 kB
RssFile:            3328 kB
RssShmem:              0 kB
VmData: 32106616 kB
VmStk:      1012 kB
VmExe:        44 kB
VmLib:     16648 kB
VmPTE:     50908 kB
VmPMD:       128 kB
VmSwap:        0 kB
HugetlbPages:          0 kB
Threads:        19922
SigQ:   0/64039
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000004
SigIgn: 0000000000000000
SigCgt: 2000000181005ecf
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000
Seccomp:        0
Speculation_Store_Bypass:       vulnerable
Cpus_allowed:   7fff
Cpus_allowed_list:      0-14
Mems_allowed:  00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list:      0
voluntary_ctxt_switches:        5986
nonvoluntary_ctxt_switches:     26

So my first question is: What is the 'Threads' number telling me? Are there 19922 threads INCLUDING the threads which are ended or are this only the currently active threads?

I also wondered why all of this threads are currently in SLEEPING state...

I made a graph (#1) which displays the current number of threads for this process and I can see that the number is not only increasing.

So why is this number so wiggly? Should the subdirectory of a thread be deleted after the thread has finished? And what is with threads with state "SLEEPING" - are they finished? Because I have nothing to wait for...

Answer 1

So, I found out that

1. the number "Threads" excludes every finished thread - so all the threads are running or waiting for something.
2. That is also the reason why this number is wiggling like that.

After checking my source code again, I found out that some ExecutorService objects are not correctly closed so I correted that and received the following graph (which looks better!)

So when somebody else have similar issues, this is what I did:

1. Login to the machine where the application is running
2. Get the correct PID from the process by running ps -aux | grep -i 'NAME' (replace NAME with the correct name of the application)

3. Get the number of running/waiting threads by executing cat /proc/[PID]/status

4. Create the graph-data with for x in {1..100000}; do echo $(date) - $(find /proc/[PID]/task -maxdepth 1 | wc -l); sleep 1; done >> thread_counter.csv