Akka Http web app fails to pass health check in Amazon ECS

Question

UPDATE

Answered by myself

So I trying to deploy my dockerized Akka Http sample app into Amazon Elastic Container Service. I'm using sbt Docker Plugin to simplify image creation.

Problem: And my app in docker works fine locally but not in EC2 instance when launched by auto-scaling group. For some reason exposed port is not accessible. Thus health check becomes "Unhealthy" and container gets killed in 15 seconds after start.

What I tried:

1. CloudWatch logs contained normal start logs:

Server online at http://0.0.0.0:4567/

2. I visited EC2 instance and tried to curl the container when it was started by agent. Result was:

curl: (7) Failed to connect to localhost port 4567: Connection refused

3. I ran docker container manually in EC2. Port only works when I run it with -p 4567:4567. But I run network mode "host" and I believe that port mapping is not required. ECS Agent doesn't use port mapping during container start (observable in docker ps).

4. I tried to deploy both host address bindings:

   Http().bindAndHandle(routes, "localhost", 4567) // doesn't work locally on Mac

   Http().bindAndHandle(routes, "0.0.0.0", 4567) // works locally in Docker

5. I tested with another docker image for comparison and it worked perfectly (tongueroo/sinatra:latest): I was able to curl it in EC2 and it passed health check.

6. ECS agent doesn't show anything useful from my point of view.

I guess something goes wrong with EXPORT 4567 or scala/java binding to port. What are your ideas?

Answer 1

It appeared to be my own fault with Scala code I copy pasted from somewhere.

OLD code which doesn't work in ECS env, but seamlessly works locally! :

val serverBinding: Future[Http.ServerBinding] = Http().bindAndHandle(routes, "0.0.0.0", 4567)

  serverBinding.onComplete {
    case Success(bound) =>
      println(s"Server online at http://${bound.localAddress.getHostString}:${bound.localAddress.getPort}/")
      StdIn.readLine() // let it run until user presses return
      serverBinding
        .flatMap(_.unbind()) // trigger unbinding from the port
        .onComplete(_ => system.terminate()) // and shutdown when done
    case Failure(e) =>
      Console.err.println(s"Server could not start!")
      e.printStackTrace()
      system.terminate()
  }

NEW fixed code which works:

  val serverBinding: Future[Http.ServerBinding] = Http().bindAndHandle(routes, "0.0.0.0", 4567)

  serverBinding.onComplete {
    case Success(bound) =>
      println(s"Server online at http://${bound.localAddress.getHostString}:${bound.localAddress.getPort}/ v8")
    case Failure(e) =>
      Console.err.println(s"Server could not start!")
      e.printStackTrace()
      system.terminate()

  println("Waiting for termination...")
  Await.result(system.whenTerminated, Duration.Inf)
  println("TERMINATED")

There were not blocking action and JVM exited quickly after binding happened.