spray-client throwing "Too many open files" exception when giving more concurrent requests

Question

I have a spray http client which is running in a server X, which will make connections to server Y. Server Y is kind of slow(will take 3+ sec for a request)

This is my http client code invocation:

def get() {
    val result = for {
       response <- IO(Http).ask(HttpRequest(GET,Uri(getUri(msg)),headers)).mapTo[HttpResponse]
    } yield response

    result onComplete {
      case Success(res) => sendSuccess(res)
      case Failure(error) => sendError(res)
    }
}

These are the configurations I have in application.conf:

spray.can {
    client {
            request-timeout = 30s
            response-chunk-aggregation-limit = 0
            max-connections = 50
            warn-on-illegal-headers = off
        }
        host-connector {
            max-connections = 128
            idle-timeout = 3s
          }
    }

Now I tried to abuse the server X with large number of concurrent requests(using ab with n=1000 and c=100).

Till 900 requests it went fine. After that the server threw lot of exceptions and I couldn't hit the server after that. These are the exceptions:

[info] [ERROR] [03/28/2015 17:33:13.276] [squbs-akka.actor.default-dispatcher-6] [akka://squbs/system/IO-TCP/selectors/$a/0] Accept error: could not accept new connection

[info] java.io.IOException: Too many open files [info] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) [info] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241) [info] at akka.io.TcpListener.acceptAllPending(TcpListener.scala:103)

and on further hitting the same server, it threw the below exception:

[info] [ERROR] [03/28/2015 17:53:16.735] [hcp-client-akka.actor.default-dispatcher-6] [akka://hcp-client/system/IO-TCP/selectors] null [info] akka.actor.ActorInitializationException: exception during creation

[info] at akka.actor.ActorInitializationException$.apply(Actor.scala:164)

[info] at akka.actor.ActorCell.create(ActorCell.scala:596)

[info] Caused by: java.lang.reflect.InvocationTargetException

[info] at sun.reflect.GeneratedConstructorAccessor59.newInstance(Unknown Source)

[info] Caused by: java.io.IOException: Too many open files [info] at sun.nio.ch.IOUtil.makePipe(Native Method)

I was previously using apache http client(which was synchronous) which was able to handle 10000+ requests with concurrency of 100.

I'm not sure I'm missing something. Any help would be appreciated.

Answer 1

The problem is that every time you call get() method it creates a new actor that creates at least one connection to the remote server. Furthermore you never shut down that actor, so each such connection leaves until it times out.

You only need a single such actor to manage all your HTTP requests, thus to fix it take IO(Http) out of the get() method and call it only once. Reuse that returned ActorRef for all your requests to that server. Shut it down on application shutdown.

For example:

val system: ActorSystem = ...
val io = IO(Http)(system)
io ! Http.Bind( ...

def get(): Unit = {
  ...
  io.ask ...
  // or
  io.tell ...
}