Search code examples
javamultithreadingcpu-usagethread-dumpdisruptor-pattern

How to analyze thread dumps in Java to minimize high CPU usage

I'm trying to read text file and insert into database with Disruptor.

But I find that the CPU usage is too high (200%, according to top command).

I'm new to performance tuning and thread dump analysis. I don't know what's going wrong.

So I execute top -H and find the two highest threads (both are 99%), and find the thread dump:

"main" prio=10 tid=0x00007f54a4006800 nid=0x79ab runnable [0x00007f54a8340000]
   java.lang.Thread.State: RUNNABLE
    at java.lang.Thread.yield(Native Method)
    at com.lmax.disruptor.SingleProducerSequencer.next(SingleProducerSequencer.java:104)
    at com.lmax.disruptor.SingleProducerSequencer.next(SingleProducerSequencer.java:79)
    at com.lmax.disruptor.RingBuffer.next(RingBuffer.java:207)
    at com.xxx.xxx.connectivity.quickfixj.FixMessageReceiver.onMessage(FixMessageReceiver.java:105)
    at com.xxx.xxx.database.DatabaseService.start(DatabaseService.java:110)
    at com.xxx.xxx.database.DatabaseService.main(DatabaseService.java:168)


"pool-2-thread-1" prio=10 tid=0x00007f54a426d800 nid=0x79bc runnable [0x00007f5492a37000]
   java.lang.Thread.State: RUNNABLE
    at java.lang.Thread.yield(Native Method)
    at com.lmax.disruptor.SingleProducerSequencer.next(SingleProducerSequencer.java:104)
    at com.lmax.disruptor.SingleProducerSequencer.next(SingleProducerSequencer.java:79)
    at com.lmax.disruptor.RingBuffer.next(RingBuffer.java:207)
    at com.cimb.reporting.connectivity.jms.DatabaseEventHandler.publish2DbRingBuffer(DatabaseEventHandler.java:49)
    at com.xxx.xxx.connectivity.jms.DatabaseEventHandler.onEvent(DatabaseEventHandler.java:39)
    at com.xxx.xxx.connectivity.jms.DatabaseEventHandler.onEvent(DatabaseEventHandler.java:15)
    at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:133)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:662)

Basically these two threads are going to publish data to Disruptor. I create Disruptor in this way:

Disruptor<TradeEvent> disruptor = new Disruptor<TradeEvent>(TradeEvent.TRADE_EVENT_FACTORY,
                properties.dbRingbufferSize(), Executors.newCachedThreadPool(),
                ProducerType.SINGLE, new BlockingWaitStrategy());

Please help me and analyze the thread dump to find the root cause of high CPU usage.

Solution

  • Faced exactly the same problem: 100% cpu usage on my machine Intel Core i3-3220 with 16GB memory with one broker (Xmx 2G), one client (Xmx2G) and without any message in a ringbuffer.

    Quick profiling shows that Thread.yield() consumes about 70-80% of cpu.

    It is turned out the YieldingWaitStrategy is not a proper strategy in my case. So in my case quick fix was to set the wait strategy into BlockingWaitStrategy:

    Disruptor<MessageEvent> disruptor = new Disruptor<MessageEvent>(eventFactory, RING_BUFFER_SIZE, executor, ProducerType.SINGLE, new BlockingWaitStrategy());

    UPDATE

    JProfiler for YieldingWaitStrategy enter image description here