Future sql execution behavior while database connection is unstable

Question

I have a problem with futures executing SQL queries.

While database connection is good futures asynchronously executing sql queries within sessions from connection pool. TypeSafe Slick helps me to get sessions from the pool.

When database connection broke down the new coming futures can't execute their queries and wait. I don't see any callbacks onComplete.

When the database connection is good again all previous futures are still waiting. Only new futures coming after reconnect can execute their work.

Please advise how to tell already called waiting futures about db reconnection and continue their work, or do callback onComplete Failure.

My configuration for c3p0 ComboPooledDataSource:

val ds = new ComboPooledDataSource
ds.setDriverClass("oracle.jdbc.OracleDriver")
ds.setJdbcUrl(jdbcUrl)
ds.setInitialPoolSize(20)
ds.setMinPoolSize(1)
ds.setMaxPoolSize(40)
ds.setAcquireIncrement(5)
ds.setMaxIdleTime(3600)

//Connection testing
ds.setTestConnectionOnCheckout(false)
ds.setTestConnectionOnCheckin(false)
ds.setIdleConnectionTestPeriod(10)

//Connection recovery
ds.setBreakAfterAcquireFailure(false)
ds.setAcquireRetryAttempts(30)
ds.setAcquireRetryDelay(10000)

val databasePool = Database.forDataSource(ds)

// Typesafe Slick session handling
def withClient[T](body: Session => T) = {
  databasePool.withSession(body)
}

Futures creates here:

class RewardActivatorHelper {

  private implicit val ec = new ExecutionContext {
    val threadPool = Executors.newFixedThreadPool(1000)
    def execute(runnable: Runnable) {threadPool.submit(runnable)}
    def reportFailure(t: Throwable) {throw t}
  }

  case class FutureResult(spStart:Long, spFinish:Long)
  def activateReward(msg:Msg, time:Long):Unit = {
  msg.users.foreach {
    user =>
      val future:Future[FutureResult] = Future {
        val (spStart, spFinish) = OracleClient.rewardActivate(user)
        FutureResult(spStart, spFinish)
    }

    future.onComplete {
      case Success(futureResult:FutureResult) =>
        //do something

      case Failure(e:Throwable) => log.error(e.getMessage())     
    }
  }
}

Answer 1

Problem solved by setTestConnectionOnCheckout(true)