Search code examples
javascalasubmitthreadpoolrunnable

Scala FixedThreadPool stops launching threads even though the previous ones have finished their job


I have a problem that has been torturing me for days

I have a FixedThreadPool with a fixed size (for simplicity let's assume 10) to which I assign 275 Runnable:

Logging.log("----- Thread Pool creation -----")
val pool = Executors.newFixedThreadPool(num_of_threads)
Logging.log("> submitting tables")
for(t <- jobTablesList) {
  val db = t(0)
  val tb = t(1)
  val st = t(2).toInt
  pool.submit(new DataMigration( db, tb, st, retry_times, file_threads))
}

pool.shutdown()
pool.awaitTermination(10000, TimeUnit.HOURS)
Logging.log("----- DATA MIGRATION END! -----")

I'm sure it's 275 because I print the length of the jobTablesList first. DataMigration class extends Runnable and that is the run method:

override def run(): Unit = {
  Logging.log("Migration start\n\ttable: " + table + "\n\tdatabase: " + database + "\n\tstatus: " + status)

  try {
    if (status == 0) createTable()
    else readTable()

    if(!isTableView) {
      if (status == 1) migrateTable()
      if (status == 2) validateTable()
    }
  }
  catch {
    case e: Throwable => migrationError(e)
  }

  Logging.log("Migration end\n\ttable:\t" + table + "\n\tdatabase:\t" + database + "\n\tstatus:\t" + status)

}

As you can see I know exactly when a thread starts and finishes its work, simply because it prints this information in the log.

The problem is that at some point the pool stops assigning tasks and while those remaining in the queue continue to be processed and completed no more are added.

for example in the last run the script got stuck in this state (num_of_threads = 15):

  • Table started: 188
  • Table ended: 188

why doesn't the pool schedule other threads? The cpu usage is very low and the ram ok

Someone can help me to understand?

Thanks!


Solution

  • After some time I figured it out...

    Basically I had DataMigration class defined in TWO DIFFERENT FILES that don't interact each other, inside TWO DIFFERENT OBJECTS (of course with different name) and Scala apparently doesn't like it but without communicating it

    It just randomly stopped submitting threads after some time

    I removed the second file and it currently continues to work in the expected way