Search code examples
grailsquartz

Quartz Jobs do not fire intermittently

Environment:

  • Application server: WebSphere 8.5
  • OS : Solaris 11
  • Quartz version: 2.2.1
  • None clustred environment.

Problem and findings so far:

  • When two concurrent jobs are scheduled to be fired, only one job fire automatically. However, this is not the case for all jobs. Sometimes, one single job also is not firing.
  • No misfire job is reported after a job is not triggered. 
[QuartzScheduler_QuartzScheduler-NON_CLUSTERED_MisfireHandler] DEBUG jdbcjobstore.JobStoreTX  - Found 0 triggers that missed their scheduled fire-time.

  • Jobs are scheduled to be run every few hours and each job won't take more than a few minutes to finish. In other words, no thread is busy which is proven in Thread Dump.

  • 30 threads are configured for Quartz. All 30 are in TIMED_WAITING state in thread dump taken before job scheduled time.

QuartzScheduler_Worker-30
*priority:3 - threadId:0x0000000104ff5000 - nativeId:0x1c1 - nativeId (decimal):449 - state:TIMED_WAITING
stackTrace:
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
- locked  (a java.lang.Object)*
  • It was working fine for a while and it happens after WAR file restart.

More details on simulated issue: - While job is not triggered based on the log. PREV_FIRE_TIME column in QUARTZ_TRIGGERS table is updated.

Solution

  • There was an issue in Database. Tablespace was full, so quartz was not able to insert records to start schedule job.

    I wonder why no Exception was thrown while there was DB issue even when I enabled the Quartz debug mode.