I have been having a lot of trouble building an application test and was hoping someone with some more Scala experience could point me in the right direction.
I have a number of data models that reside in a Postgres database, mapped to case classes using Slick. My Play app then provides JSON based REST endpoints to said data models. Since most of the actual code is similar between each endpoint, most of the code is implemented as a trait which is mixed in to the actual controllers which over-ride the necessary bits.
This works fine, but when I try to run unit tests on each of them most controllers work then I end up with the error:
[error] Can't find a constructor for class helpers.DatabaseHelper
[warn] c.z.h.HikariConfig - The jdbcConnectionTest property is now deprecated, see the documentation for connectionTestQuery
[error]
[error] cannot create an instance for class FileControllerSpec
[error] caused by java.sql.SQLTransientConnectionException: db - Connection is not available, request timed out after 1005ms.
[error] caused by org.postgresql.util.PSQLException: FATAL: remaining connection slots are reserved for non-replication superuser connections
[error]
[error] STACKTRACE
[error] sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[error] sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[error] sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[error] java.lang.reflect.Constructor.newInstance(Constructor.java:423)
[error] org.specs2.reflect.Classes$$anonfun$org$specs2$reflect$Classes$$createInstanceForConstructor$1.apply(Classes.scala:104)
[error] org.specs2.control.ActionT$$anonfun$safe$1.apply(ActionT.scala:88)
[error] org.specs2.control.ActionT$$anonfun$reader$1$$anonfun$apply$6.apply(ActionT.scala:79)
[error] org.specs2.control.Status$.safe(Status.scala:100)
[error] org.specs2.control.StatusT$$anonfun$safe$1.apply(StatusT.scala:62)
[error] org.specs2.control.StatusT$$anonfun$safe$1.apply(StatusT.scala:62)
[error] scalaz.syntax.ToApplicativeOps$$anon$1.self$lzycompute(ApplicativeSyntax.scala:29)
[error] scalaz.syntax.ToApplicativeOps$$anon$1.self(ApplicativeSyntax.scala:29)
[error] scalaz.syntax.ToApplicativeOps$ApplicativeIdV$$anonfun$point$1.apply(ApplicativeSyntax.scala:33)
[error] scalaz.WriterTApplicative$$anonfun$point$1.apply(WriterT.scala:282)
[error] scalaz.WriterTApplicative$$anonfun$point$1.apply(WriterT.scala:282)
[error] scalaz.effect.IO$$anonfun$apply$19$$anonfun$apply$20.apply(IO.scala:136)
[error] scalaz.effect.IO$$anonfun$apply$19$$anonfun$apply$20.apply(IO.scala:136)
[error] scalaz.FreeFunctions$$anonfun$return_$1.apply(Free.scala:326)
[error] scalaz.FreeFunctions$$anonfun$return_$1.apply(Free.scala:326)
[error] scalaz.std.FunctionInstances$$anon$1$$anonfun$map$1.apply(Function.scala:56)
[error] scalaz.Free$$anonfun$run$1.apply(Free.scala:172)
[error] scalaz.Free$$anonfun$run$1.apply(Free.scala:172)
[error] scalaz.Free.go2$1(Free.scala:119)
[error] scalaz.Free.go(Free.scala:122)
[error] scalaz.Free.run(Free.scala:172)
[error] scalaz.effect.IO$class.unsafePerformIO(IO.scala:22)
[error] scalaz.effect.IOFunctions$$anon$6.unsafePerformIO(IO.scala:227)
[error] org.specs2.reflect.Classes$$anonfun$createInstance$1$$anonfun$apply$1$$anonfun$3.apply(Classes.scala:37)
[error] org.specs2.reflect.Classes$$anonfun$createInstance$1$$anonfun$apply$1$$anonfun$3.apply(Classes.scala:36)
[error] scala.collection.immutable.List.map(List.scala:273)
[error] org.specs2.reflect.Classes$$anonfun$createInstance$1$$anonfun$apply$1.apply(Classes.scala:36)
[error] org.specs2.reflect.Classes$$anonfun$createInstance$1$$anonfun$apply$1.apply(Classes.scala:29)
[error] scala.Function1$$anonfun$andThen$1.apply(Function1.scala:52)
[error] org.specs2.control.Status$class.fold(Status.scala:30)
[error] org.specs2.control.Ok.fold(Status.scala:95)
[error] org.specs2.control.Status$class.flatMap(Status.scala:48)
[error] org.specs2.control.Ok.flatMap(Status.scala:95)
[error] org.specs2.control.Status$class.map(Status.scala:45)
[error] org.specs2.control.Ok.map(Status.scala:95)
[error] org.specs2.control.StatusT$$anonfun$map$1.apply(StatusT.scala:16)
[error] org.specs2.control.StatusT$$anonfun$map$1.apply(StatusT.scala:16)
[error] scalaz.WriterT$$anonfun$map$1.apply(WriterT.scala:46)
[error] scalaz.WriterT$$anonfun$map$1.apply(WriterT.scala:46)
[error] scalaz.effect.IO$$anonfun$map$1$$anonfun$apply$8.apply(IO.scala:56)
[error] scalaz.effect.IO$$anonfun$map$1$$anonfun$apply$8.apply(IO.scala:55)
[error] scalaz.Free$$anonfun$map$1.apply(Free.scala:52)
[error] scalaz.Free$$anonfun$map$1.apply(Free.scala:52)
[error] scalaz.Free$$anonfun$flatMap$1$$anonfun$apply$1.apply(Free.scala:60)
[error] scalaz.Free$$anonfun$flatMap$1$$anonfun$apply$1.apply(Free.scala:60)
[error] scalaz.Free.resume(Free.scala:72)
[error] scalaz.Free.go2$1(Free.scala:118)
[error] scalaz.Free.go(Free.scala:122)
[error] scalaz.Free.run(Free.scala:172)
[error] scalaz.effect.IO$class.unsafePerformIO(IO.scala:22)
[error] scalaz.effect.IOFunctions$$anon$6.unsafePerformIO(IO.scala:227)
[error] org.specs2.runner.SbtRunner$$anonfun$newTask$1$$anon$4.execute(SbtRunner.scala:37)
[error] sbt.ForkMain$Run$2.call(ForkMain.java:294)
[error] sbt.ForkMain$Run$2.call(ForkMain.java:284)
[error] java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error] java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[error] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[error] java.lang.Thread.run(Thread.java:745)
DatabaseHelper is an object that sets up my test data.
Reducing the number of specs makes the error go away, so I know that the issue is not with that particular test spec.
I am using runtime DI as recommended by the Play documentation and over-riding the database binding for testing to avoid blanking my development database.
I am using Play evolutions to manage the database schema, but intentionally blanking the database and re-setting it up during setup/teardown in order to ensure that it is pristine before each test spec.
I think that what is going on is that all of my test specs are being initialised at the same time, which means that they all try to connect to the database at the same time and it therefore runs out of connections.
I have tried using the parallelExecution and concurrentRestrictions settings in sbt to set only one process going at a time but this is to no avail. I've tried setting each spec to run sequentially as well but this seems not to work. I have also tried catching the exception and retrying the setup but this seems not to work either.
I am now at a loss as to what to try to make my tests work! Please help.
Thanks a lot.
Test case:
@RunWith(classOf[JUnitRunner])
class FileControllerSpec extends GenericControllerSpec {
sequential
override val componentName: String = "FileController"
override val uriRoot: String = "/file"
override def testParsedJsonObject(checkdata: JsLookupResult, parsed_test_json: JsValue) = {
val object_keys = Seq("filepath","user","ctime","mtime","atime")
val object_keys_int = Seq("storage","version")
object_keys.map(key=>
(checkdata \ key).as[String] must equalTo((parsed_test_json \ key).as[String])
) ++ object_keys_int.map(key=>
(checkdata \ key).as[Int] must equalTo((parsed_test_json \ key).as[Int])
)
}
override val testGetId: Int = 3
override val testGetDocument: String = """{"filepath":"/path/to/a/video.mxf","storage":1,"user":"me","version":1,"ctime":"1970-01-01T04:25:45.678+0100","mtime":"1970-01-01T04:25:45.678+0100","atime":"1970-01-01T04:25:45.678+0100"}"""
override val testCreateDocument: String = """{"filepath":"/path/to/some/other.project","storage":1,"user":"test","version":3,"ctime":"2017-03-17T13:51:00.123+0000","mtime":"2017-03-17T13:51:00.123+0000","atime":"2017-03-17T13:51:00.123+0000"}"""
override val minimumNewRecordId = 3
override val testDeleteId: Int = 2
override val testConflictId: Int = -1
}
GenericControllerSpec:
@RunWith(classOf[JUnitRunner])
trait GenericControllerSpec extends Specification with BeforeAfterAll {
//can over-ride bindings here. see https://www.playframework.com/documentation/2.5.x/ScalaTestingWithGuice
val application:Application = new GuiceApplicationBuilder()
.overrides(bind[DatabaseConfigProvider].to[TestDatabase.testDbProvider])
.build
val injector:Injector = new GuiceApplicationBuilder()
.overrides(bind[DatabaseConfigProvider].to[TestDatabase.testDbProvider])
.injector()
def inject[T : ClassTag]: T = injector.instanceOf[T]
//needed for body.consumeData
implicit val system = ActorSystem("storage-controller-spec")
implicit val materializer = ActorMaterializer()
protected val databaseHelper:DatabaseHelper = inject[DatabaseHelper]
val logger: Logger = Logger(this.getClass)
override def beforeAll(): Unit ={
logger.warn(">>>> before all <<<<")
val theFuture = databaseHelper.setUpDB().map({
case Success(result)=>logger.info("DB setup successful")
case Failure(error)=>logger.error(s"DB setup failed: $error")
})
Await.result(theFuture, 10.seconds)
}
override def afterAll(): Unit ={
logger.warn("<<<< after all >>>>")
Await.result(databaseHelper.teardownDB(), 10.seconds)
}
val componentName:String
val uriRoot:String
def testParsedJsonObject(checkdata:JsLookupResult,test_parsed_json:JsValue):Seq[MatchResult[Any]]
val testGetId:Int
val testGetDocument:String
val testCreateDocument:String
val testDeleteId:Int
val testConflictId:Int
val minimumNewRecordId:Int
def bodyAsJsonFuture(response:Future[play.api.mvc.Result]) = response.flatMap(result=>
result.body.consumeData.map(contentBytes=> {
logger.debug(contentBytes.decodeString("UTF-8"))
Json.parse(contentBytes.decodeString("UTF-8"))
}
)
)
componentName should {
"return 400 on a bad request" in {
logger.debug(s"$uriRoot/boum")
val response = route(application,FakeRequest(GET, s"$uriRoot/boum")).get
status(response) must equalTo(BAD_REQUEST)
}
"return valid data for a valid record" in {
logger.warn(s"Test URL is $uriRoot/1")
val response:Future[play.api.mvc.Result] = route(application, FakeRequest(GET, s"$uriRoot/1")).get
status(response) must equalTo(OK)
val jsondata = Await.result(bodyAsJsonFuture(response), 5.seconds).as[JsValue]
(jsondata \ "status").as[String] must equalTo("ok")
(jsondata \ "result" \ "id").as[Int] must equalTo(1)
testParsedJsonObject(jsondata \ "result", Json.parse(testGetDocument))
}
"accept new data to create a new record" in {
val response = route(application, FakeRequest(
method="PUT",
uri=uriRoot,
headers=FakeHeaders(Seq(("Content-Type", "application/json"))),
body=testCreateDocument)
).get
status(response) must equalTo(OK)
val jsondata = Await.result(bodyAsJsonFuture(response), 5.seconds).as[JsValue]
(jsondata \ "status").as[String] must equalTo("ok")
(jsondata \ "detail").as[String] must equalTo("added")
(jsondata \ "id").as[Int] must greaterThanOrEqualTo(minimumNewRecordId) //if we re-run the tests without blanking the database explicitly this goes up
val newRecordId = (jsondata \ "id").as[Int]
val checkResponse = route(application, FakeRequest(GET, s"$uriRoot/$newRecordId")).get
val checkdata = Await.result(bodyAsJsonFuture(checkResponse), 5.seconds)
(checkdata \ "status").as[String] must equalTo("ok")
(checkdata \ "result" \ "id").as[Int] must equalTo(newRecordId)
testParsedJsonObject(checkdata \ "result", Json.parse(testCreateDocument))
}
"delete a record" in {
val response = route(application, FakeRequest(
method="DELETE",
uri=s"$uriRoot/$testDeleteId",
headers=FakeHeaders(),
body="")
).get
status(response) must equalTo(OK)
val jsondata = Await.result(bodyAsJsonFuture(response), 5.seconds).as[JsValue]
(jsondata \ "status").as[String] must equalTo("ok")
(jsondata \ "detail").as[String] must equalTo("deleted")
(jsondata \ "id").as[Int] must equalTo(testDeleteId)
}
"return conflict (409) if attempting to delete something with sub-objects" in {
val response = route(application, FakeRequest(
method = "DELETE",
uri = s"$uriRoot/$testConflictId",
headers = FakeHeaders(),
body = "")
).get
status(response) must equalTo(CONFLICT)
val jsondata = Await.result(bodyAsJsonFuture(response), 5.seconds).as[JsValue]
(jsondata \ "status").as[String] must equalTo("error")
(jsondata \ "detail").as[String] must equalTo("This is still referenced by sub-objects")
}
}
}
DatabaseHelper:
class DatabaseHelper @Inject()(configuration: Configuration, dbConfigProvider: DatabaseConfigProvider) {
private val dbConfig = dbConfigProvider.get[JdbcProfile]
private val logger: Logger = Logger(this.getClass)
def setUpDB():Future[Try[Unit]] = {
logger.warn("In setUpDB")
dbConfig.db.run(
DBIO.seq(
(TableQuery[FileAssociationRow].schema ++
TableQuery[FileEntryRow].schema ++
TableQuery[ProjectEntryRow].schema ++
TableQuery[ProjectTemplateRow].schema ++
TableQuery[ProjectTypeRow].schema ++
TableQuery[StorageEntryRow].schema
).create,
TableQuery[StorageEntryRow] += StorageEntry(None,None,"filesystem",Some("me"),None,None,None),
TableQuery[StorageEntryRow] += StorageEntry(None,None,"omms",Some("you"),None,None,None),
TableQuery[FileEntryRow] += FileEntry(None,"/path/to/a/video.mxf",1,"me",1,new Timestamp(12345678),new Timestamp(12345678),new Timestamp(12345678)),
TableQuery[FileEntryRow] += FileEntry(None,"/path/to/secondtestfile",1,"tstuser",1,new Timestamp(123456789),new Timestamp(123456789),new Timestamp(123456789)),
//"""{"name": "Premiere test template 1","projectTypeId": 1,"filepath", "storageId": 1}"""
//"{"name":,"opensWith":"AdobePremierePro.app","targetVersion":"14.0"}"
TableQuery[ProjectTypeRow] += ProjectType(None,"Premiere 2014 test","AdobePremierePro.app","14.0"),
TableQuery[ProjectTypeRow] += ProjectType(None,"Cubase 7.0 test","Cubase.app","7.0"),
TableQuery[ProjectTemplateRow] += ProjectTemplate(Some(1),"Premiere test template 1",1,"/srv/projectfiles/ProjectTemplatesDev/Premiere/premiere_template_2014.prproj",1)
).asTry
)
}
def teardownDB():Future[Try[Unit]] = {
logger.warn("In teardownDB")
dbConfig.db.run(
DBIO.seq(
(
TableQuery[FileAssociationRow].schema ++
TableQuery[FileEntryRow].schema ++
TableQuery[ProjectEntryRow].schema ++
TableQuery[ProjectTemplateRow].schema ++
TableQuery[ProjectTypeRow].schema ++
TableQuery[StorageEntryRow].schema
).drop
).asTry
)
}
}
build.sbt settings:
concurrentRestrictions in Global := Seq(
Tags.limit(Tags.Test, 1),
Tags.limitAll(1)
)
parallelExecution in Test := false
the exception tells you that your connection pool is out of connections.
I see two issues here:
Due to performance reasons I recommend you anyways to use H2DB drivers with postgres settings for unit testing as long you are not using postgres specific features that H2DB is not able to emulate.