Search code examples
spring-bootspring-batchscheduler

Spring Batch Job scheduler stops being executing after several runs

On a project we have to run a job that starts periodically (every 5 minutes on QA env now) that processes some assignments for 40k users. We had decided to utilize Spring Batch because it fits perfectly and implemented it with pretty much default configuration (e.g. it uses SyncTaskExecutor under the hood). Okay, so, there is a job that consists of a single step with:

  1. out-of-box HibernatePagingItemReader
  2. custom ItemProcessorthat performs lightweight calculations in memory
  3. custom ItemWriter that persists data to the same PostgreSQL db via several JPQL and native queries.

The job itself is scheduled with @EnableScheduling and is being triggered every 5 mins by cron expression:

@Scheduled(cron = "${job.assignment-rules}")
void processAssignments() {
  try {
    log.debug("Running assignment processing job");
    jobLauncher.run(assignmentProcessingJob, populateJobParameters());
  } catch (JobExecutionException e) {
    log.error("Job processing has failed", e);
  }
}

Here is a cron expression from application.yml:

job:
  assignment-rules: "0 0/5  * * * *"

The problem is that it stops being scheduled after several runs (different amount of runs every time). Let's take a look into the Spring Batch schema:

select ex.job_instance_id, ex.create_time, ex.start_time, ex.end_time, ex.status, ex.exit_code, ex.exit_message
from batch_job_execution ex inner join batch_job_instance bji on ex.job_instance_id = bji.job_instance_id
order by start_time desc, job_instance_id desc;

enter image description here And then silence. Nothing special in logs. The only thing I believe could make sense is that there are two more jobs running on that instance. And one of them is time consuming because it sends emails via SMTP. The entire jobs schedule is:

job:
  invitation-email: "0 0/10 * * * *"
  assignment-rules: "0 0/5  * * * *"
  rm-subordinates-count: "0 0/30 * * * *"

Colleagues, could anybody point me out the way this problem could be troubleshooted?

Thanks a lot in advance

Solution

  • Using the default SyncTaskExecutor to launch jobs is not safe in your use case as all jobs will be executed by a single thread. If one of the jobs takes more than 5 minutes to run, next jobs will pile up and fail to start at some point.

    I would configure a JobLauncher with an asynchronous TaskExecutor implementation (like org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor) in your use case. You can find an example in the Configuring a JobLauncher section (See "Figure 3. Asynchronous Job Launcher Sequence").

    Apart from the task executor configured to be used by the JobLauncher of Spring Batch, you need to make sure you have the right task executor used by Spring Boot to schedule tasks (since you are using @EnableScheduling). Please refer to the Task Execution and Scheduling section for more details.