Search code examples
jmsactivemq-artemismessagebroker

Decreased performance in ActiveMQ Artemis after paging

After the broker switched to paging mode I am seeing a strange drop in performance. Some messages began to take a very long time: 1800мс 10мс 15мс 700мс

I am also seeing a lot of disk usage:

enter image description here

My broker.xml:

<configuration>
   <core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="urn:activemq:core ">
         
      <thread-pool-max-size>50</thread-pool-max-size>
      <name>0.0.0.0</name>


      <persistence-enabled>true</persistence-enabled>

      <journal-type>ASYNCIO</journal-type>

      <paging-directory>data/paging</paging-directory>

      <bindings-directory>data/bindings</bindings-directory>

      <journal-directory>data/journal</journal-directory>

      <large-messages-directory>data/large-messages</large-messages-directory>

      <journal-datasync>true</journal-datasync>

      <journal-min-files>2</journal-min-files>

      <journal-pool-files>10</journal-pool-files>

      <journal-file-size>10M</journal-file-size>

      <journal-buffer-timeout>16000</journal-buffer-timeout>

      <journal-max-io>4096</journal-max-io>
      
      <disk-scan-period>5000</disk-scan-period>

      <max-disk-usage>90</max-disk-usage>

      <critical-analyzer>true</critical-analyzer>

      <critical-analyzer-timeout>120000</critical-analyzer-timeout>

      <critical-analyzer-check-period>60000</critical-analyzer-check-period>

      <critical-analyzer-policy>HALT</critical-analyzer-policy>


      <acceptors>

         <acceptor name="artemis">tcp://0.0.0.0:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=CORE,AMQP;useEpoll=true;</acceptor>

         <acceptor name="amqp">tcp://0.0.0.0:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpLowCredits=300</acceptor>

      </acceptors>
   </core>
</configuration>

Linux Astra, 4 CPU 24GB ram 50GB SSD, ActiveMQ Artemis 2.7.0

only broker restart helps

Solution

  • A decrease in performance is expected when paging. This is because messages are being paged to and from disk instead of being accessed directly from RAM. Even the fastest disks are much slower than RAM therefore paging reduces performance.

    There are a few ways to mitigate this performance decrease:

    • Provide the broker's JVM with enough heap space so that paging never occurs.
    • Use flow-control to prevent the excessive build-up of messages on the broker that leads to paging.
    • Ensure that message consumption keeps up with message production to prevent the excessive build-up of messages on the broker that leads to paging (e.g. add more consumers, increase performance of existing consumers, etc.).
    • Use high-speed SSDs instead of slower traditional HDDs.

    My guess is that you're using mostly non-durable messages so that restarting the broker clears out these messages and eliminates the need to page thus restoring normal performance.

    Also, since your using ActiveMQ Artemis 2.7.0 I strongly recommend you upgrade to the latest release. It's been over 2 years now since 2.7.0 was released and there have been many bug fixes and new features implemented in later versions.