I have a fairly large Play Framework application running Play 2.4.3 in production. I've had a few instances where suddenly all requests to the server start getting the following MatchError:
! @6p4o8d49h - Internal server error, for (GET) [/bookings/desk/suggest-products?term=&maxSize=50&page=1] ->
play.api.UnexpectedException: Unexpected exception[MatchError: null]
at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:261) ~[com.typesafe.play.play_2.11-2.4.3.jar:2.4.3]
at play.api.http.HttpErrorHandlerExceptions.throwableToUsefulException(HttpErrorHandler.scala) ~[com.typesafe.play.play_2.11-2.4.3.jar:2.4.3]
at play.http.DefaultHttpErrorHandler.throwableToUsefulException(DefaultHttpErrorHandler.java:169) ~[com.typesafe.play.play_2.11-2.4.3.jar:2.4.3]
at play.http.DefaultHttpErrorHandler.onServerError(DefaultHttpErrorHandler.java:131) ~[com.typesafe.play.play_2.11-2.4.3.jar:2.4.3]
at play.core.j.JavaHttpErrorHandlerAdapter$$anonfun$onServerError$1.apply(JavaHttpErrorHandlerAdapter.scala:22) [com.typesafe.play.play_2.11-2.4.3.jar:2.4.3]
at play.core.j.JavaHttpErrorHandlerAdapter$$anonfun$onServerError$1.apply(JavaHttpErrorHandlerAdapter.scala:22) [com.typesafe.play.play_2.11-2.4.3.jar:2.4.3]
at play.core.j.JavaHelpers$$anonfun$invokeWithContext$1.apply(JavaHelpers.scala:140) [com.typesafe.play.play_2.11-2.4.3.jar:2.4.3]
at play.core.j.JavaHelpers$$anonfun$invokeWithContext$1.apply(JavaHelpers.scala:139) [com.typesafe.play.play_2.11-2.4.3.jar:2.4.3]
at play.core.j.JavaHelpers$class.withContext(JavaHelpers.scala:151) [com.typesafe.play.play_2.11-2.4.3.jar:2.4.3]
at play.core.j.JavaHelpers$.withContext(JavaHelpers.scala:160) [com.typesafe.play.play_2.11-2.4.3.jar:2.4.3]
at play.core.j.JavaHelpers$class.invokeWithContext(JavaHelpers.scala:139) [com.typesafe.play.play_2.11-2.4.3.jar:2.4.3]
at play.core.j.JavaHelpers$.invokeWithContext(JavaHelpers.scala:160) [com.typesafe.play.play_2.11-2.4.3.jar:2.4.3]
at play.core.j.JavaHttpErrorHandlerAdapter.onServerError(JavaHttpErrorHandlerAdapter.scala:22) [com.typesafe.play.play_2.11-2.4.3.jar:2.4.3]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$9$$anonfun$apply$1.applyOrElse(PlayDefaultUpstreamHandler.scala:158) [com.typesafe.play.play-netty-server_2.11-2.4.3.jar:2.4.3]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$9$$anonfun$apply$1.applyOrElse(PlayDefaultUpstreamHandler.scala:155) [com.typesafe.play.play-netty-server_2.11-2.4.3.jar:2.4.3]
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36) [org.scala-lang.scala-library-2.11.7.jar:na]
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:216) [org.scala-lang.scala-library-2.11.7.jar:na]
at scala.util.Try$.apply(Try.scala:192) [org.scala-lang.scala-library-2.11.7.jar:na]
at scala.util.Failure.recover(Try.scala:216) [org.scala-lang.scala-library-2.11.7.jar:na]
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324) [org.scala-lang.scala-library-2.11.7.jar:na]
at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:324) [org.scala-lang.scala-library-2.11.7.jar:na]
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) [org.scala-lang.scala-library-2.11.7.jar:na]
at play.api.libs.iteratee.Execution$trampoline$.executeScheduled(Execution.scala:109) [com.typesafe.play.play-iteratees_2.11-2.4.3.jar:2.4.3]
at play.api.libs.iteratee.Execution$trampoline$.execute(Execution.scala:71) [com.typesafe.play.play-iteratees_2.11-2.4.3.jar:2.4.3]
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) [org.scala-lang.scala-library-2.11.7.jar:na]
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248) [org.scala-lang.scala-library-2.11.7.jar:na]
at scala.concurrent.Promise$class.complete(Promise.scala:55) [org.scala-lang.scala-library-2.11.7.jar:na]
at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153) [org.scala-lang.scala-library-2.11.7.jar:na]
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:23) [org.scala-lang.scala-library-2.11.7.jar:na]
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39) [com.typesafe.akka.akka-actor_2.11-2.4.0.jar:na]
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:399) [com.typesafe.akka.akka-actor_2.11-2.4.0.jar:na]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [org.scala-lang.scala-library-2.11.7.jar:na]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [org.scala-lang.scala-library-2.11.7.jar:na]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [org.scala-lang.scala-library-2.11.7.jar:na]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [org.scala-lang.scala-library-2.11.7.jar:na]
Caused by: scala.MatchError: null
at play.core.j.JavaAction.apply(JavaAction.scala:97) ~[com.typesafe.play.play_2.11-2.4.3.jar:2.4.3]
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:105) ~[com.typesafe.play.play_2.11-2.4.3.jar:2.4.3]
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:105) ~[com.typesafe.play.play_2.11-2.4.3.jar:2.4.3]
at play.utils.Threads$.withContextClassLoader(Threads.scala:21) ~[com.typesafe.play.play_2.11-2.4.3.jar:2.4.3]
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:104) ~[com.typesafe.play.play_2.11-2.4.3.jar:2.4.3]
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:103) ~[com.typesafe.play.play_2.11-2.4.3.jar:2.4.3]
at scala.Option.map(Option.scala:146) ~[org.scala-lang.scala-library-2.11.7.jar:na]
at play.api.mvc.Action$$anonfun$apply$1.apply(Action.scala:103) ~[com.typesafe.play.play_2.11-2.4.3.jar:2.4.3]
at play.api.mvc.Action$$anonfun$apply$1.apply(Action.scala:96) ~[com.typesafe.play.play_2.11-2.4.3.jar:2.4.3]
at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:524) ~[com.typesafe.play.play-iteratees_2.11-2.4.3.jar:2.4.3]
at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:524) ~[com.typesafe.play.play-iteratees_2.11-2.4.3.jar:2.4.3]
at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:560) ~[com.typesafe.play.play-iteratees_2.11-2.4.3.jar:2.4.3]
at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:560) ~[com.typesafe.play.play-iteratees_2.11-2.4.3.jar:2.4.3]
at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$13.apply(Iteratee.scala:536) ~[com.typesafe.play.play-iteratees_2.11-2.4.3.jar:2.4.3]
at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$13.apply(Iteratee.scala:536) ~[com.typesafe.play.play-iteratees_2.11-2.4.3.jar:2.4.3]
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) [org.scala-lang.scala-library-2.11.7.jar:na]
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) [org.scala-lang.scala-library-2.11.7.jar:na]
... 6 common frames omitted
Before the first error occurs the logs look quite healthy and normal and the first exception is some random request. After the first request to hit this error, all subsequent requests get the same error until we restart the server.
From looking at play.core.j.JavaAction.apply(JavaAction.scala:97) it looks like the error means the returned Response objects are all null.
I looked at some graphs of the heap space usage from the time when this happened and there is a huge spike in memory usage. I strongly suspect the MatchErrors are hiding OutOfMemoryError("Java heap space") errors.
Is there a way I can stop the scala.MatchError from hiding the actual OutOfMemoryError?
This is caused by a combination of the heap being full, and Scala needing to allocate heap to do pattern matching.
The solution is to fix whatever is causing the OutOfMemoryException, the MatchError is a red herring. It's just a side effect of running out of heap space.