Search code examples
javaspringhibernatescheduled-tasksbulk

Correct way of doing bulk operations through scheduled taks?


In my application, I have set of scheduled tasks which process all users' data and perform various tasks like earning, tax calculations, generate statements for all users. Since these processes need to run for each user, they take a lot of time(many hours) because of large number of users. Data processing for one user is completely independent from another user's, so they can be run in parallel. What are my options here? What are the best practices for doing such large/bulk operations. We are using J2SE platform with spring, jpa and hibernate.


Solution

  • You should be doing the same via batch.

    Since you have mentioned that you were doing with Spring you can consider using Spring Batch.

    Spring Batch provides reusable functions that are essential in processing large volumes of records, including logging/tracing, transaction management, job processing statistics, job restart, skip, and resource management. It also provides more advanced technical services and features that will enable extremely high-volume and high performance batch jobs through optimization and partitioning techniques.

    Check out the Reference manual on how to implement.