I have a Scala application running in a Tomcat7 that lately has started randomly crashing due to ClassNotFoundException errors for standard classes such as scala.collection.immutable.ListSet even when the server has been running perfectly fine for some time. The errors started getting more frequent and now we're getting several crashes a day on some servers
This is a sample of the crash log we're getting in the catalina.out log file.
Dec 25, 2018 8:50:47 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8080"]
Dec 25, 2018 8:50:47 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 6938 ms
Uncaught error from thread [TPDispatcher-akka.actor.default-dispatcher-4] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[TPDispatcher]
java.lang.NoClassDefFoundError: scala/collection/immutable/ListSet$
at scala.collection.immutable.HashSet$HashSet1.updated0(HashSet.scala:145)
at scala.collection.immutable.HashSet$HashTrieSet.updated0(HashSet.scala:227)
at scala.collection.immutable.HashSet$HashTrieSet.updated0(HashSet.scala:227)
at scala.collection.immutable.HashSet$HashTrieSet.updated0(HashSet.scala:227)
at scala.collection.immutable.HashSet$HashTrieSet.updated0(HashSet.scala:227)
at scala.collection.immutable.HashSet.$plus(HashSet.scala:56)
at scala.collection.immutable.HashSet.$plus(HashSet.scala:33)
at scala.collection.mutable.SetBuilder.$plus$eq(SetBuilder.scala:24)
at scala.collection.mutable.SetBuilder.$plus$eq(SetBuilder.scala:22)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.HashSet$HashSet1.foreach(HashSet.scala:153)
at scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:306)
at scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:306)
at scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:306)
at scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:306)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)
at scala.collection.SetLike$class.map(SetLike.scala:93)
at scala.collection.AbstractSet.map(Set.scala:47)
at com.mycompany.lib.actors.BadgeUpdater.updateBadge(BadgeUpdater.scala:23)
at com.mycompany.lib.actors.BadgeUpdater$$anonfun$receive$1.applyOrElse(BadgeUpdater.scala:15)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at com.twinpush.tp_dispatcher.lib.actors.BadgeUpdater.aroundReceive(BadgeUpdater.scala:11)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.ClassNotFoundException: scala.collection.immutable.ListSet$
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1859)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1702)
... 33 more
Dec 26, 2018 2:15:31 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-8080"]
Dec 26, 2018 2:15:31 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
I've tried recompiling several times, with no luck. I have no idea of what may cause ClassNotFoundException on a server that's already running, so any hint will be highly appreciated.
Environment info:
Scala version: 2.10.4
Server version: Apache Tomcat/7.0.68 (Ubuntu)
Server built: Oct 30 2018 12:54:52 UTC
Server number: 7.0.68.0
OS Name: Linux
OS Version: 4.4.0-141-generic
Architecture: amd64
JVM Version: 1.8.0_191-8u191-b12-0ubuntu0.16.04.1-b12
JVM Vendor: Oracle Corporation
Turns out I had a library that was leaking file descriptors and at some point the JVM was unable to open class files (is that even possible?). Upgrading the library fixed the leak and the JVM stopped crashing.