Our current application was running on glassfish and windows server. We moved the application to jetty and running on linux server now. there is a quartz job in the application which runs every minute and getting ftp files from remote server. Recently I realized that the job stopped suddenly and there is no exception associated with quartz.
I added a quartz.properties file and configured it as below,
org.quartz.scheduler.skipUpdateCheck = true
org.quartz.scheduler.instanceName = MyQuartzScheduler
org.quartz.scheduler.jobFactory.class = org.quartz.simpl.SimpleJobFactory
org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount=10
org.quartz.scheduler.jmx.export = true
org.quartz.scheduler.threadName="QuartzThreadFa"
I also realized the thread dump after quartz stopped to working (application is running without any error)
on the thread dump I see the quartz is using all threads until thread count end. And all threads sems locked by an object,
Thread Dump;
"qtp1074009022-11344" prio=10 tid=0x00007f480c00e000 nid=0x2dd0 waiting on condition [0x00007f4783ffe000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e6b313d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:390)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:509)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:563)
at java.lang.Thread.run(Thread.java:745)
"Timer-1" daemon prio=10 tid=0x00007f47c8007000 nid=0x2bd0 in Object.wait() [0x00007f47838f3000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000000e7187828> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"Timer-0" daemon prio=10 tid=0x00007f477c01e000 nid=0x2bb6 in Object.wait() [0x00007f4783efd000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000000e70b1aa0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"Scheduler-1583944291" prio=10 tid=0x00007f47f8008000 nid=0x2bb4 waiting on condition [0x00007f48542af000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e6c7cdf8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"DestroyJavaVM" prio=10 tid=0x00007f488800d000 nid=0x2b34 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"qtp1074009022-49-acceptor-3-ServerConnector@cd9f9e{HTTP/1.1}{0.0.0.0:10900}" prio=10 tid=0x00007f478c001000 nid=0x2bb1 waiting for monitor entry [0x00007f48543b0000]
java.lang.Thread.State: BLOCKED (on object monitor)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:225)
- waiting to lock <0x00000000e70e29c8> (a java.lang.Object)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:321)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:460)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532)
at java.lang.Thread.run(Thread.java:745)
"qtp1074009022-48-acceptor-2-ServerConnector@cd9f9e{HTTP/1.1}{0.0.0.0:10900}" prio=10 tid=0x00007f47cc001000 nid=0x2bb0 waiting for monitor entry [0x00007f48544b1000]
java.lang.Thread.State: BLOCKED (on object monitor)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:225)
- waiting to lock <0x00000000e70e29c8> (a java.lang.Object)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:321)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:460)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532)
at java.lang.Thread.run(Thread.java:745)
"QuartzThreadFa" prio=10 tid=0x00007f4888747800 nid=0x2bad in Object.wait() [0x00007f48547bf000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool.blockForAvailableThreads(SimpleThreadPool.java:452)
- locked <0x00000000e6e2f0c8> (a java.lang.Object)
at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:263)
"MyQuartzScheduler_Worker-10" prio=10 tid=0x00007f488873e000 nid=0x2bac runnable [0x00007f48548c0000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:832)
at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:759)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3293)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3271)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3171)
at com.fatr.myApp.trawler.jobs.FtpJob.getRemoteFiles(FtpJob.java:85)
at com.fatr.myApp.trawler.jobs.FtpJob.execute(FtpJob.java:48)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
- locked <0x00000000e6c80e58> (a java.lang.Object)
"MyQuartzScheduler_Worker-9" prio=10 tid=0x00007f488873c800 nid=0x2bab runnable [0x00007f48549c1000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:832)
at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:759)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3293)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3271)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3171)
at com.fatr.myApp.trawler.jobs.FtpJob.getRemoteFiles(FtpJob.java:85)
at com.fatr.myApp.trawler.jobs.FtpJob.execute(FtpJob.java:48)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
- locked <0x00000000e6d80280> (a java.lang.Object)
"MyQuartzScheduler_Worker-8" prio=10 tid=0x00007f4888730000 nid=0x2baa runnable [0x00007f4854ac2000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:832)
at org.apache.commons.net.ftp.FTPClient._retrieveFile(FTPClient.java:1854)
at org.apache.commons.net.ftp.FTPClient.retrieveFile(FTPClient.java:1845)
at com.fatr.myApp.trawler.jobs.FtpJob.createFileAndDownloadFromFTP(FtpJob.java:209)
at com.fatr.myApp.trawler.jobs.FtpJob.getRemoteFiles(FtpJob.java:126)
at com.fatr.myApp.trawler.jobs.FtpJob.execute(FtpJob.java:48)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
- locked <0x00000000e6d80298> (a java.lang.Object)
"MyQuartzScheduler_Worker-7" prio=10 tid=0x00007f488872e000 nid=0x2ba9 runnable [0x00007f4854bc3000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:832)
at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:759)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3293)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3271)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3171)
at com.fatr.myApp.trawler.jobs.FtpJob.getRemoteFiles(FtpJob.java:85)
at com.fatr.myApp.trawler.jobs.FtpJob.execute(FtpJob.java:48)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
- locked <0x00000000e6d8e180> (a java.lang.Object)
"MyQuartzScheduler_Worker-6" prio=10 tid=0x00007f488872c000 nid=0x2ba8 runnable [0x00007f4854cc3000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:832)
at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:759)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3293)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3271)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3171)
at com.fatr.myApp.trawler.jobs.FtpJob.getRemoteFiles(FtpJob.java:85)
at com.fatr.myApp.trawler.jobs.FtpJob.execute(FtpJob.java:48)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
- locked <0x00000000e6d802b0> (a java.lang.Object)
"MyQuartzScheduler_Worker-5" prio=10 tid=0x00007f4888729800 nid=0x2ba7 runnable [0x00007f4854dc4000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:832)
at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:759)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3293)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3271)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3171)
at com.fatr.myApp.trawler.jobs.FtpJob.getRemoteFiles(FtpJob.java:85)
at com.fatr.myApp.trawler.jobs.FtpJob.execute(FtpJob.java:48)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
- locked <0x00000000e6d802c8> (a java.lang.Object)
"MyQuartzScheduler_Worker-4" prio=10 tid=0x00007f4888727800 nid=0x2ba6 runnable [0x00007f4854ec6000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:832)
at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:759)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3293)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3271)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3171)
at com.fatr.myApp.trawler.jobs.FtpJob.getRemoteFiles(FtpJob.java:85)
at com.fatr.myApp.trawler.jobs.FtpJob.execute(FtpJob.java:48)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
- locked <0x00000000e6d802e0> (a java.lang.Object)
"MyQuartzScheduler_Worker-3" prio=10 tid=0x00007f4888726000 nid=0x2ba5 runnable [0x00007f4854fc7000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:832)
at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:759)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3293)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3271)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3171)
at com.fatr.myApp.trawler.jobs.FtpJob.getRemoteFiles(FtpJob.java:85)
at com.fatr.myApp.trawler.jobs.FtpJob.execute(FtpJob.java:48)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
- locked <0x00000000e6d802f8> (a java.lang.Object)
"MyQuartzScheduler_Worker-2" prio=10 tid=0x00007f4888724800 nid=0x2ba4 runnable [0x00007f48550c8000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:832)
at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:759)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3293)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3271)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3171)
at com.fatr.myApp.trawler.jobs.FtpJob.getRemoteFiles(FtpJob.java:85)
at com.fatr.myApp.trawler.jobs.FtpJob.execute(FtpJob.java:48)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
- locked <0x00000000e6c80e70> (a java.lang.Object)
"MyQuartzScheduler_Worker-1" prio=10 tid=0x00007f4888720800 nid=0x2ba3 runnable [0x00007f48551c9000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:832)
at org.apache.commons.net.ftp.FTPClient._retrieveFile(FTPClient.java:1854)
at org.apache.commons.net.ftp.FTPClient.retrieveFile(FTPClient.java:1845)
at com.fatr.myApp.trawler.jobs.FtpJob.createFileAndDownloadFromFTP(FtpJob.java:209)
at com.fatr.myApp.trawler.jobs.FtpJob.getRemoteFiles(FtpJob.java:118)
at com.fatr.myApp.trawler.jobs.FtpJob.execute(FtpJob.java:48)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
- locked <0x00000000e6d80310> (a java.lang.Object)
"qtp1074009022-32-acceptor-1-ServerConnector@cd9f9e{HTTP/1.1}{0.0.0.0:10900}" prio=10 tid=0x00007f4888584800 nid=0x2b77 runnable [0x00007f48555dc000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
- locked <0x00000000e70e29c8> (a java.lang.Object)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:321)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:460)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532)
at java.lang.Thread.run(Thread.java:745)
"qtp1074009022-31-acceptor-0-ServerConnector@cd9f9e{HTTP/1.1}{0.0.0.0:10900}" prio=10 tid=0x00007f4888582800 nid=0x2b76 waiting for monitor entry [0x00007f48556dd000]
java.lang.Thread.State: BLOCKED (on object monitor)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:225)
- waiting to lock <0x00000000e70e29c8> (a java.lang.Object)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:321)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:460)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532)
at java.lang.Thread.run(Thread.java:745)
"qtp1074009022-30-selector-7" prio=10 tid=0x00007f4888580800 nid=0x2b75 runnable [0x00007f48557de000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000e70b1ad0> (a sun.nio.ch.Util$2)
- locked <0x00000000e70b1ae8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e705e970> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:452)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:428)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532)
at java.lang.Thread.run(Thread.java:745)
"qtp1074009022-29-selector-6" prio=10 tid=0x00007f488857e800 nid=0x2b74 runnable [0x00007f48558df000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000e70e6478> (a sun.nio.ch.Util$2)
- locked <0x00000000e70e6490> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e6e48fe8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:452)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:428)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532)
at java.lang.Thread.run(Thread.java:745)
"qtp1074009022-28-selector-5" prio=10 tid=0x00007f488857c800 nid=0x2b73 runnable [0x00007f48559e0000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000e70e2a70> (a sun.nio.ch.Util$2)
- locked <0x00000000e70e2a88> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e70c5f80> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:452)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:428)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532)
at java.lang.Thread.run(Thread.java:745)
"qtp1074009022-27-selector-4" prio=10 tid=0x00007f488857a000 nid=0x2b72 runnable [0x00007f4855ae1000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000e7091b38> (a sun.nio.ch.Util$2)
- locked <0x00000000e7091b50> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e70ac7e8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:452)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:428)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532)
at java.lang.Thread.run(Thread.java:745)
"qtp1074009022-26-selector-3" prio=10 tid=0x00007f4888578000 nid=0x2b71 runnable [0x00007f4855be2000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000e70e64c0> (a sun.nio.ch.Util$2)
- locked <0x00000000e70e64d8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e6e48f58> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:452)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:428)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532)
at java.lang.Thread.run(Thread.java:745)
"qtp1074009022-25-selector-2" prio=10 tid=0x00007f4888576800 nid=0x2b70 runnable [0x00007f4855ce3000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000e70e6508> (a sun.nio.ch.Util$2)
- locked <0x00000000e70e6520> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e7039d10> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:452)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:428)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532)
at java.lang.Thread.run(Thread.java:745)
"qtp1074009022-24-selector-1" prio=10 tid=0x00007f4888575000 nid=0x2b6f runnable [0x00007f4855de4000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000e70e2ab8> (a sun.nio.ch.Util$2)
- locked <0x00000000e70e2ad0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e70c5e18> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:452)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:428)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532)
at java.lang.Thread.run(Thread.java:745)
"qtp1074009022-23-selector-0" prio=10 tid=0x00007f4888571000 nid=0x2b6e runnable [0x00007f4855ee5000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000e70e2b00> (a sun.nio.ch.Util$2)
- locked <0x00000000e70e2b18> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e70c5ea8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:452)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:428)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532)
at java.lang.Thread.run(Thread.java:745)
"org.eclipse.jetty.util.RolloverFileOutputStream" daemon prio=10 tid=0x00007f48884ea800 nid=0x2b4c in Object.wait() [0x00007f48569f6000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000000e6b8cc48> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
The problem is that the application was running for some years without any quartz issue on Glassfish /Windows, Now it just running until thread count ends on Jetty/Linux and it is not able to stop/kill threads. any help would be appreciated, thank you
This issue looks like is an issue between FTPClient(on lx servers) and FTPServer. FTPClient from Linux hangs after certain point of time. Because of unusual nature of communication between FTPClient(in this case its Linux) and server certain port numbers should be open on Client(Linux Server) for FTPServer to send back data. But due to firewall restrictions most of these unprivileged ports are closed on Linux. Apparently it looks like FTPServer uses a different port each time a request comes from FTPClient, some of which are not open on Linux.
The solution looks like FTPClient to enter "PASSIVE" mode. In passive mode FTPClient(on Linux) opens a dedicated port for server to send back data.
So based on this information I have set FTPClient to enter passive and deployed again.
FtpClient.enterLocalPassiveMode()
it is working now without any issue.