Search code examples
javamysqlspringtomcatcamunda

Camunda DB connection closed but picked by camunda engine

We are using camunda with RDS/MySql as DB. It works fine but then sometimes it says DB is closed and so throws the ProcessEngine Exception. Here is what I understood from our config and logs:

  1. We have 5active connections at any time in our pool (Specified in datasource config)

  2. There was a scenario where it was closed.

  3. We saw error like:

Request received Context path: /engine-rest Request received Path

Info: /user PathInfo: /user ExceptionHandler:

org.camunda.bpm.engine.ProcessEngineException: Process engine

persistence exception at

org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.rethrow(CommandInvocationContext.java:148)

    at org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:173)

    at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:113)

    at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:66)

    at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:30)

...... Caused by: org.apache.ibatis.exceptions.PersistenceException:

Error querying database. Cause:

com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:

No operations allowed after connection closed. The error may exist in

org/camunda/bpm/engine/impl/mapping/entity/User.xml The error may involve

org.camunda.bpm.engine.impl.persistence.entity.UserEntity.selectUserByQueryCriteria

The error occurred while executing a query SQL: select distinct RES.*

from ACT_ID_USER RES

order by RES.ID_ asc LIMIT ? OFFSET ? Cause:

com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:

No operations allowed after connection closed.

Our tomcat props specify: minIdle = 5;

My best guess: Its closed on server but we are maintaining locally due to the above property.

Per tomcat doc (https://tomcat.apache.org/tomcat-8.0-doc/jdbc-pool.html):

testOnBorrow = true; validationQuery = "select 1";

The two props should fix it as it validates the connection.

Question I am trying to figure out:

  1. How can I repro this issue? Apart from keeping connection it idle for several hours, the scenario where this happened.

  2. Does the AWS RDS server close the connection? If so, can we control it?

Solution

  • As mentioned by @Zelldon, the connection timeout can be reduced and then we could try it. It works as expected.

    Just to be sure, I ran two instances of camunda, one with this fix and other without. Could see that the fix worked.

    Regarding RDS, it does close the connection but I could not find any documentation on it.