Search code examples
hikaricp

Reconnect HikariCP after DB reset


This is similar to this question I posted about 2 years ago.

I have a simple test which continuously reads from the database. I stop/start my SQL Server instance.

@Test
public void testConnectionReset() {
while (true) {
  try {
   simpleRead();
  }
  catch (Exception e) {
    e.printStackStrace();
  }
}

@com.google.inject.persist.Transactional
protected void simpleRead() {
  dao = new MyDao(....)
  dao.findBy(...)
}

If the error occurs in simpleRead(), all is handled correctly. If the error occurs in testConnectionReset(), I have the following stackTrace and no reconnect when the database is restarted.

[2017-Dec-21 10:47:26] - [ERROR] - I/O Error: Connection reset by peer: socket write error
    org.hibernate.TransactionException: Unable to rollback against JDBC Connection
        at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.rollback(AbstractLogicalConnectionImplementor.java:121)
        at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.rollback(JdbcResourceLocalTransactionCoordinatorImpl.java:250)
        at org.hibernate.engine.transaction.internal.TransactionImpl.rollback(TransactionImpl.java:100)
        at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.rollbackIfNecessary(JpaLocalTxnInterceptor.java:153)
        at com.google.inject.persist.jpa.JpaLocalTxnInterceptor.invoke(JpaLocalTxnInterceptor.java:74)
        at temp.testConnection(TestConnection.java:45)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
        at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
        at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
        at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
        at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
        at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
        at org.testng.TestRunner.privateRun(TestRunner.java:767)
        at org.testng.TestRunner.run(TestRunner.java:617)
        at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
        at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:329)
        at org.testng.SuiteRunner.privateRun(SuiteRunner.java:291)
        at org.testng.SuiteRunner.run(SuiteRunner.java:240)
        at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
        at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
        at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
        at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
        at org.testng.TestNG.run(TestNG.java:1057)
        at org.testng.remote.AbstractRemoteTestNG.run(AbstractRemoteTestNG.java:132)
        at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:236)
        at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:81)
    Caused by: java.sql.SQLException: Connection is closed
        at com.zaxxer.hikari.pool.ProxyConnection$ClosedConnection.lambda$getClosedConnection$0(ProxyConnection.java:490)
        at com.sun.proxy.$Proxy84.rollback(Unknown Source)
        at com.zaxxer.hikari.pool.ProxyConnection.rollback(ProxyConnection.java:377)
        at com.zaxxer.hikari.pool.HikariProxyConnection.rollback(HikariProxyConnection.java)
        at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.rollback(AbstractLogicalConnectionImplementor.java:116)
        ... 29 more

My workaround for the moment is to set com.zaxxer.hikari.aliveBypassWindowMsto 0, forcing a connection check every time before it is returned by the pool. I find it hard to believe that this is a Hikari bug since I dont see anyone else with this problem so perhaps its some other configuration I am missing.

My stack is

  • List item
  • SQL Server
  • JTDS (1.3.1) driver. (I have tried the SQL Server driver with similar results)
  • HikariCP (2.7.4)
  • Google guice (4.1.0)
  • Hibernate (5.2.11)

Solution

  • This is a bug in guice-persist. It is very closely related to this one. The simple flow of events is something like this

    1. Hikari has a valid connection in the pool
    2. Connection to the DB is lost
    3. Guice-prersist @Transactional is called, and pulls connection from pool (unchecked).
    4. Guice executes txn.begin(), throws exception BUT fails to end unitOfWork / closeConnection. Pool remains in bad state.

    The workarounds I have found are one of the following

    1. Handle the unitOfWork manually.
    2. Ask Hikari to check the connection every time its returned from the pool by setting aliveBypassWindowMs to 0.
    3. Modify the offending class (JpaLocalTxnInterceptor) directly by ending the unitOfWork when necessary.