Improving Akka Remote Throughput

Question

We're thinking about using Akka for client server communications, and trying to benchmark data transfer. Currently we're trying to send a million messages where each message is a case class with 8 string fields.

At this point we're struggling to get acceptable performance. We see about 600KB/s transfer rate and idle CPUs on client and server, so something is going wrong. Maybe it's our netty configuration.

This is our akka config

Server {
  akka {
    extensions = ["akka.contrib.pattern.ClusterReceptionistExtension"]
    loggers = ["akka.event.slf4j.Slf4jLogger"]
    loglevel = "DEBUG"
    logging-filter = "akka.event.slf4j.Slf4jLoggingFilter"
    log-dead-letters = 10
    log-dead-letters-during-shutdown = on
    actor {
      provider = "akka.cluster.ClusterActorRefProvider"
    }
    remote {
      enabled-transports = ["akka.remote.netty.tcp"]
      netty.tcp {
        hostname = "instance.myserver.com"
        port = 2553
        maximum-frame-size = 1000 MiB
        send-buffer-size = 2000 MiB
        receive-buffer-size = 2000 MiB
      }
    }
    cluster {
      seed-nodes = ["akka.tcp://[email protected]:2553"]

      roles = [master]
    }
    contrib.cluster.receptionist {
      name = receptionist
      role = "master"
      number-of-contacts = 3
      response-tunnel-receive-timeout = 300s
    }
  }

}

Client {
  akka {
    loggers = ["akka.event.slf4j.Slf4jLogger"]
    loglevel = "DEBUG"
    logging-filter = "akka.event.slf4j.Slf4jLoggingFilter"
    actor {
      provider = "akka.remote.RemoteActorRefProvider"
    }
    remote {
      enabled-transports = ["akka.remote.netty.tcp"]
      netty.tcp {
        hostname = "127.0.0.1"
        port = 0
        maximum-frame-size = 1000 MiB
        send-buffer-size = 2000 MiB
        receive-buffer-size = 2000 MiB
      }
    }
    cluster-client {
      initial-contacts = ["akka.tcp://[email protected]:2553/user/receptionist"]
      establishing-get-contacts-interval = "10s"
      refresh-contacts-interval = "10s"
    }
  }

}

UPDATE:

In the end, notwithstanding the discussion about serialisation (see below) we just converted our payloads to use byte arrays, that way serialisation would not affect the tests. We found that on core i7 with 24gb ram using jeroMQ (ie zeroMQ re-implemented in java - so still not the fastest) we saw consistently about 200k msgs/sec or about 20 MB/sec whilst on raw akka (ie without zeroMQ plugin) we saw about 10k msgs/sec or just under 1MB/sec. Trying on akka + zeroMQ made the performance worse.

Answer 1

To make it easy to get started with remoting akka uses Java serialization for message serialization, this is not what you'd typically use in production because it's not very fast and does not handle message versioning well.

What you should do is use kryo or protobuf for serialization and you should be able to get much better numbers.

You can read about how here, there are also a few links to available serializers at the bottom of the page: http://doc.akka.io/docs/akka/current/scala/serialization.html