Search code examples
javasqloracle-databasehibernatejpa

How to process large amount data using Hibernate

I am using hibernate for processing data in my application. Application is working fine but i am facing time related performance in application. The scenario is, i have one table that is located remotely and contain around 100000 rows. i have to insert that data in local database table(with different structure) using some mapping(so that i can know which remote table column is equivalent to local table column). it is taking 9 hours for processing that data. I am executing native SQL queries. is it causing performance issue? Any suggestion will be appreciated.

Solution

    1. Set the following Hibernate properties to enable batching:

      <property name="hibernate.order_updates" value="true"/>
<property name="hibernate.order_inserts" value="true"/>
<property name="hibernate.jdbc.batch_versioned_data" value="true"/>
<property name="hibernate.jdbc.fetch_size" value="50"/>
<property name="hibernate.jdbc.batch_size" value="50"/>
<property name="hibernate.default_batch_fetch_size" value="50"/>

    2. You need to clear the Session once a batch is processed to clear memory. This allows you to use a smaller Heap size, therefore reducing the chance of long GC runs:

      session.flush();
session.clear();

    3. Use the new identifier generators and in case you use DB sequences you can choose the pooled-lo optimizer. Using a hi/lo algorithm will reduce sequence calls and increase performance.

    4. Don't use the identity generator, because that's going to disable batching