Search code examples
hibernategoogle-cloud-dataflowapache-beamhikaricp

Google Cloud dataflow : How to initialize Hikari connection pool only once per worker (singleton)?


Hibernate Utils is creating the session factory along with Hikari configuration. Currently we are doing inside @Setup method of ParDo, but it opens way too many connections. So is there any good example to initialize connection pool per worker ?


Solution

  • If you are using @Setup method inside DoFn to create a database connection keep in mind that Apache Beam would create connection pool per worker instance thread. This might result in a lot of database connections depending on the number of workers that spin up.

    You can use a Singleton class inside a ParDo to create your connection pool and control the number of connections per worker. You can take a look here for a sample implementation

    http://mail-archives.apache.org/mod_mbox/beam-issues/201905.mbox/%3CJIRA.13231789.1557156230000.210182.1557222420198@Atlassian.JIRA%3E