How to update/restart database connection with Panache Hibernate

Question

I have my secrets in AWS Secrets Manager. And some quarkus API's running behind an ECS cluster, which connects to an RDS Database. The secrets are pulled at start time (or initialization) of the API's application and the secrets of the RDS rotate every week which leads connection errors during runtime with the following errors:

org.hibernate.engine.jdbc.spi.SqlExceptionHelper
FATAL: password authentication failed for user
avax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection

My application is written in Kotlin, uses quarkus as a framework and Hibernate ORM with Panache to connect to the RDS. The application uses an application .yml as a config source and a custom data source config class, indicated under org.eclipse.microprofile.config.spi.ConfigSource, which gets the secrets from AWS and overrides getValue method and sets some datasource configs like password during initialization (start time). The database is an Aurora PostgreSQL but I guess that is irrelevant.

I would like the application at run time to be able to establish a connection with the database using the correct password at runtime. Preferably still using Panache, but if there is not other way I could change the database connection library. How could the connection get the right credentials at runtime? How to tell Panache to open a new connection with some new credentials?

Another solution to my issue could be to restart the ECS tasks. But the tasks definitions seems to remember the old connection credentials. And I don't want to create a new task definition every week. How could I restart the application correctly?

I have tried adding a custom credential provider (using this as a reference https://quarkus.io/guides/credentials-provider). But it only runs during initalization. See config below (appended in application.yml):

    datasource:
      credentials-provider: custom
      credentials-provider-name: custom-credentials-provider

But it is only executed at start time. So I still have the same issue.

I have also read Setting properties programmatically in Hibernate and UnsupportedOperationException: The application must supply JDBC connections. But they use different config source (hibernate.cfg.configsource) and they don't use Panache.

I have also tried to set the new password at runtime. I injected the password

    @ConfigProperty(name = "quarkus.datasource.password")
    lateinit var password: String

    ... //  I fetch the new passord and change it.

But it also does not work, as it only modifies the copy of the config file. Will Panache re-establish a new connection with the right credentials If I manage to change the config source?

Answer 1

I made it work adding a custom credential provider like in here https://quarkus.io/guides/credentials-provider.

My mistake is that we use different profiles for dev and prod and I forgot to add the configs in production in the application.yml. You can add it to you production profile like this:

"%prod":
  quarkus:
    datasource:
      credentials-provider: custom
      credentials-provider-name: custom-credentials-provider

An example of custom credential provider:

package <WRITE YOUR PACKAGE HERE>

import io.quarkus.credentials.CredentialsProvider
import io.quarkus.credentials.CredentialsProvider.PASSWORD_PROPERTY_NAME
import io.quarkus.credentials.CredentialsProvider.USER_PROPERTY_NAME
import javax.enterprise.context.ApplicationScoped
import javax.inject.Inject
import javax.inject.Named

@ApplicationScoped
@Named("custom-credentials-provider")
class CustomCredentialsProvider : CredentialsProvider {

    @Inject
    private lateinit var config: DataSourceConfigSource

    override fun getCredentials(credentialsProviderName: String): Map<String, String?> {
        return mutableMapOf(
            Pair(PASSWORD_PROPERTY_NAME, <Add your logic to get the secret from the vault>),
            Pair(USER_PROPERTY_NAME, <Add your logic to get the secret from the vault>)
        )
    }
}

Why does this solution work with the secret rotation? After debugging the hibernate connection, you can see that hibernate is releasing each jdbc connection after each transaction. Thus, every new transaction creates a new connection pulling the updated secrets with the custom credential provider. This might bring extra network or vault manager costs (AWS secrets manager in my case). Using IAM authentication instead of password authentication might be a more cost efficient option since you could maintain a token for longer periods of time https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.html