Search code examples
javamysqlmariadbinnodbhikaricp

How to investigate active MySQL transactions that are not visible in processlist?


We have a Java application that works with MariaDB 10.4.28 hosted in AWS RDS.

We have a situation that application went down with reason "CannotCreateTransactionException". The pool has 40 active connections and 0 idle connections.

We have several tools that fire at such moments to help debug such situations:

  • ThreadStackDump
    • The dump shows that there are no actual threads running any queries at all, all threads are waiting for a connection from a pool.
  • "show processlist"
    • It shows no queries running except just a couple normal ones that are not hanged and finish within milliseconds
  • "SHOW ENGINE INNODB STATUS"
    • It in fact shows a lot of records like the following
---TRANSACTION 2974372519, ACTIVE 31 sec
  2 lock struct(s), heap size 1128, 0 row lock(s), undo log entries 4

Questions:

  • What can be a command or an approach to understand that this 2974372519 transaction is, what it is attempting to do?
  • What can be an approach to understand what my 40 occupied DB pool connections are doing?

We use HikariCP:4.0.3 and mysql-connector-java:8.0.22.


Solution

  • Transactions are stored in INFORMATION_SCHEMA.INNODB_TRX. To find the statement, just execute

    SELECT trx_query FROM INFORMATION_SCHEMA.INNODB_TRX WHERE trx_id=2974372519;
    

    See also: INFORMATION_SCHEMA.INNODB_TRX table