Our Glassfish 3.12 went to 100% last night, which locked the users out of the system (there are only two). I did a dump of the Glassfish log (below) and there are ton of locks, although none of them used more than 1 second of CPU. 100% CPU usage seems to be a common problem with Glassfish 3.1.2, but I haven't found a clear solution yet. Any ideas welcome.
./asadmin generate-jvm-report --type=thread --user admin
Thread Synchronization Statistics:
-----------------------
Number of times this thread was blocked (to enter/reenter a Monitor): 12
Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 0
Total CPU time for this thread: 0 seconds 79,691,017 nanoseconds.
User-level CPU time for this thread: 0 seconds 79,691,017 nanoseconds.
Object Monitors currently held or requested by this thread: []
Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
--------------------------------------------------------------------------------
Thread Execution Information:
-----------------------
Thread "Thread-12" thread-id: 24 thread-state: TIMED_WAITING Waiting on lock: java.util.concurrent.atomic.AtomicInteger@4cc6afaa
at: java.lang.Object.wait(Native Method)
at: com.sun.grizzly.Controller.waitUntilSelectorHandlersStop(Controller.java:1119)
at: com.sun.grizzly.Controller.start(Controller.java:869)
at: com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1285)
at: com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1123)
at: com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1212)
at: com.sun.enterprise.v3.services.impl.GrizzlyListener.startEndpoint(GrizzlyListener.java:142)
at: com.sun.enterprise.v3.services.impl.GrizzlyProxy$2.run(GrizzlyProxy.java:342)
Thread Synchronization Statistics:
-----------------------
Number of times this thread was blocked (to enter/reenter a Monitor): 0
Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 946
Total CPU time for this thread: 0 seconds 17,073,572 nanoseconds.
User-level CPU time for this thread: 0 seconds 17,073,572 nanoseconds.
Object Monitors currently held or requested by this thread: []
Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
--------------------------------------------------------------------------------
Thread Execution Information:
-----------------------
Thread "Grizzly-kernel-thread(1)" thread-id: 22 thread-state: RUNNABLE Running in native
at: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at: sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at: sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at: sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
at: sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at: com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:513)
at: com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:190)
at: com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:132)
at: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at: java.lang.Thread.run(Thread.java:745)
Thread Synchronization Statistics:
-----------------------
Number of times this thread was blocked (to enter/reenter a Monitor): 1
Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 0
Total CPU time for this thread: 0 seconds 55,218,828 nanoseconds.
User-level CPU time for this thread: 0 seconds 55,218,828 nanoseconds.
Object Monitors currently held or requested by this thread: []
Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
--------------------------------------------------------------------------------
Thread Execution Information:
-----------------------
Thread "Grizzly-kernel-thread(1)" thread-id: 20 thread-state: RUNNABLE Running in native
at: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at: sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at: sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at: sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
at: sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at: com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:513)
at: com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:190)
at: com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:132)
at: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at: java.lang.Thread.run(Thread.java:745)
Thread Synchronization Statistics:
-----------------------
Number of times this thread was blocked (to enter/reenter a Monitor): 3
Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 0
Total CPU time for this thread: 0 seconds 60,298,518 nanoseconds.
User-level CPU time for this thread: 0 seconds 60,298,518 nanoseconds.
Object Monitors currently held or requested by this thread: []
Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
--------------------------------------------------------------------------------
Thread Execution Information:
-----------------------
Thread "Thread-7" thread-id: 19 thread-state: TIMED_WAITING Waiting on lock: java.util.concurrent.atomic.AtomicInteger@836e947
at: java.lang.Object.wait(Native Method)
at: com.sun.grizzly.Controller.waitUntilSelectorHandlersStop(Controller.java:1119)
at: com.sun.grizzly.Controller.start(Controller.java:869)
at: com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1285)
at: com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1123)
at: com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1212)
at: com.sun.enterprise.v3.services.impl.GrizzlyListener.startEndpoint(GrizzlyListener.java:142)
at: com.sun.enterprise.v3.services.impl.GrizzlyProxy$2.run(GrizzlyProxy.java:342)
Thread Synchronization Statistics:
-----------------------
Number of times this thread was blocked (to enter/reenter a Monitor): 0
Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 946
Total CPU time for this thread: 0 seconds 21,716,796 nanoseconds.
User-level CPU time for this thread: 0 seconds 21,716,796 nanoseconds.
Object Monitors currently held or requested by this thread: []
Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
--------------------------------------------------------------------------------
Thread Execution Information:
-----------------------
Thread "Thread-4" thread-id: 16 thread-state: TIMED_WAITING Waiting on lock: java.util.concurrent.atomic.AtomicInteger@4d0d3887
at: java.lang.Object.wait(Native Method)
at: com.sun.grizzly.Controller.waitUntilSelectorHandlersStop(Controller.java:1119)
at: com.sun.grizzly.Controller.start(Controller.java:869)
at: com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1285)
at: com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1123)
at: com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1212)
at: com.sun.enterprise.v3.services.impl.GrizzlyListener.startEndpoint(GrizzlyListener.java:142)
at: com.sun.enterprise.v3.services.impl.GrizzlyProxy$2.run(GrizzlyProxy.java:342)
Thread Synchronization Statistics:
-----------------------
Number of times this thread was blocked (to enter/reenter a Monitor): 2
Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 946
Total CPU time for this thread: 0 seconds 58,339,427 nanoseconds.
User-level CPU time for this thread: 0 seconds 58,339,427 nanoseconds.
Object Monitors currently held or requested by this thread: []
Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
--------------------------------------------------------------------------------
Thread Execution Information:
-----------------------
Thread "Thread-2" thread-id: 14 thread-state: WAITING Waiting on lock: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1efa4e96
at: sun.misc.Unsafe.park(Native Method)
at: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at: java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:374)
at: com.sun.enterprise.server.logging.GFFileHandler.log(GFFileHandler.java:665)
at: com.sun.enterprise.server.logging.GFFileHandler$1.run(GFFileHandler.java:179)
Thread Synchronization Statistics:
-----------------------
Number of times this thread was blocked (to enter/reenter a Monitor): 0
Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 78
Total CPU time for this thread: 0 seconds 68,044,526 nanoseconds.
User-level CPU time for this thread: 0 seconds 68,044,526 nanoseconds.
Object Monitors currently held or requested by this thread: []
Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
--------------------------------------------------------------------------------
Thread Execution Information:
-----------------------
Thread "pool-1-thread-1" thread-id: 13 thread-state: TIMED_WAITING Waiting on lock: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@433d69bd
at: sun.misc.Unsafe.park(Native Method)
at: java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
at: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
at: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at: java.lang.Thread.run(Thread.java:745)
Thread Synchronization Statistics:
-----------------------
Number of times this thread was blocked (to enter/reenter a Monitor): 0
Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 1,892
Total CPU time for this thread: 0 seconds 210,072,724 nanoseconds.
User-level CPU time for this thread: 0 seconds 210,072,724 nanoseconds.
Object Monitors currently held or requested by this thread: []
Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
--------------------------------------------------------------------------------
Thread Execution Information:
-----------------------
Thread "FelixStartLevel" thread-id: 12 thread-state: WAITING Waiting on lock: java.util.ArrayList@74e919c7
at: java.lang.Object.wait(Native Method)
at: java.lang.Object.wait(Object.java:503)
at: org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:272)
at: java.lang.Thread.run(Thread.java:745)
Thread Synchronization Statistics:
-----------------------
Number of times this thread was blocked (to enter/reenter a Monitor): 42
Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 42
Total CPU time for this thread: 0 seconds 688,900,829 nanoseconds.
User-level CPU time for this thread: 0 seconds 688,900,829 nanoseconds.
Object Monitors currently held or requested by this thread: []
Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
--------------------------------------------------------------------------------
Thread Execution Information:
-----------------------
Thread "FelixDispatchQueue" thread-id: 11 thread-state: WAITING Waiting on lock: java.util.ArrayList@739b93c7
at: java.lang.Object.wait(Native Method)
at: java.lang.Object.wait(Object.java:503)
at: org.apache.felix.framework.util.EventDispatcher.run(EventDispatcher.java:1063)
at: org.apache.felix.framework.util.EventDispatcher.access$000(EventDispatcher.java:54)
at: org.apache.felix.framework.util.EventDispatcher$1.run(EventDispatcher.java:101)
at: java.lang.Thread.run(Thread.java:745)
Thread Synchronization Statistics:
-----------------------
Number of times this thread was blocked (to enter/reenter a Monitor): 232
Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 217
Total CPU time for this thread: 0 seconds 117,033,893 nanoseconds.
User-level CPU time for this thread: 0 seconds 117,033,893 nanoseconds.
Object Monitors currently held or requested by this thread: []
Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
--------------------------------------------------------------------------------
Thread Execution Information:
-----------------------
Thread "Signal Dispatcher" thread-id: 4 thread-state: RUNNABLE
Thread Synchronization Statistics:
-----------------------
Number of times this thread was blocked (to enter/reenter a Monitor): 0
Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 0
Total CPU time for this thread: 0 seconds 70,212 nanoseconds.
User-level CPU time for this thread: 0 seconds 70,212 nanoseconds.
Object Monitors currently held or requested by this thread: []
Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
--------------------------------------------------------------------------------
Thread Execution Information:
-----------------------
Thread "Finalizer" thread-id: 3 thread-state: WAITING Waiting on lock: java.lang.ref.ReferenceQueue$Lock@6b03291c
at: java.lang.Object.wait(Native Method)
at: java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
at: java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at: java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
Thread Synchronization Statistics:
-----------------------
Number of times this thread was blocked (to enter/reenter a Monitor): 20
Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 19
Total CPU time for this thread: 0 seconds 52,757,964 nanoseconds.
User-level CPU time for this thread: 0 seconds 52,757,964 nanoseconds.
Object Monitors currently held or requested by this thread: []
Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
--------------------------------------------------------------------------------
Thread Execution Information:
-----------------------
Thread "Reference Handler" thread-id: 2 thread-state: WAITING Waiting on lock: java.lang.ref.Reference$Lock@1c802427
at: java.lang.Object.wait(Native Method)
at: java.lang.Object.wait(Object.java:503)
at: java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
Thread Synchronization Statistics:
-----------------------
Number of times this thread was blocked (to enter/reenter a Monitor): 18
Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 18
Total CPU time for this thread: 0 seconds 39,254,957 nanoseconds.
User-level CPU time for this thread: 0 seconds 39,254,957 nanoseconds.
Object Monitors currently held or requested by this thread: []
Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []
No deadlock found
I found in the log an OutOfMemory our due to garbage collection. I increased some of the JVM parameters in the Glassfish server and set it to "server" instead of "client" and it seems to be ok.