I have a multithreaded Java server where I am monitoring how many threads it is using. Recently, I have noticed that OS reports many more threads for the app than JMX. For example:
cat /sys/fs/cgroup/pids/system.slice/NAME.service/pids.current
294
ps -eLf | grep java | grep NAME | wc
294
curl -s localhost:9102 | grep jvm_threads_current
223.0
Where port 9102 hosts Prometheus JMX agent. So we see the difference is 71 thread - the OS sees 71 extra thread compared to JMX. While I can assume Java might run some worker threads not reported in JVM/JMX stats, 70+ threads sounds like a lot. Is there something wrong here? Are those two measuring different things? Is there a Prometheus bug maybe? Or these threads are normal and if so - what exactly they are doing?
JVM version is 1.8.0_162-8u162-b12-1~deb9u1-b12 running on Linux.
Recently, I have noticed that OS reports many more threads for the app than JMX
In writing a simple main program, I see the same. JMX says that I have 12 threads (main, GC, finalizer, JMX, ...) while ps -eLf says I have 29. I've never noticed this discrepancy probably because I typically use the ps -a instead of -e style command.
Is there something wrong here? Are those two measuring different things? Is there a Prometheus bug maybe? Or these threads are normal and if so - what exactly they are doing?
I think you a peering behind the veil at how the OS administers the Java threads and shouldn't worry about. The number of processes used by the OS versus Java threads is going to depend highly on your version of Java and the OS native thread implementations. I don't think they are "doing" anything but are most likely there for some OS accounting reason that you don't need to concern yourself about.