Poor performance with Rack::Cache

I have Rack::Cache set up as the very top (last) middleware in my stack. My app is hosted on heroku cedar, so Rack::Cache is responsible for static assets as well.

It's backed by the heroku memcached product, and configured like this:

config.action_dispatch.rack_cache = {
  :metastore    => Dalli::Client.new,
  :entitystore  => 'file:tmp/cache/rack/body',
  :allow_reload => false
}
config.static_cache_control = "public, max-age=2592000"

It's running on thin.

I'm testing performance on an image file, using ab.

ab -n100 -c10 https://example.com/foo.jpg

Looking in my logs, all the requests are a cache hit ("fresh"). But the performance is poor, only around 6 requests/sec. I know that Rack::Cache isn't going to perform as well as a dedicated http proxy, but I certainly expected higher throughput than this.

update I don't how the heck I didn't think of this before, but my benchmark is indeed maxing out my internet connection. If I do the same test with robots.txt instead of a jpg, I get 20 requests/sec (and still max out the downstream).

Solution

I noticed the URL in your post is HTTPS. I had an issue on the cedar stack with hostname-based SSL where certain requests (for both static assets and otherwise) would experience high latency even though the logs said the queue time was 0ms and the service time was 5-10ms. Here's Heroku's support response (what he doesn't mention is that varnish doesn't cache resources coming from cedar apps, regardless, despite it being present in the stack when running hostname based SSL):

We're not entirely sure what is causing this, but we suspect Varnish is to blame. Normally, with the Cedar stack, Varnish is not present. But when the SSL Hostname add-on is used, Varnish is added back into the response process. There's no need to use both Rack::Cache and Varnish.

I'd like to suggest removing Varnish from the picture to see if it helps. The way to do this is to move SSL to our new SSL add-on, SSL Endpoint. SSL Endpoint is technically still in beta, but it will be generally available soon and it will cost the same as SSL Hostname. It's the future of SSL on Heroku and it's better in every way.

http://devcenter.heroku.com/articles/ssl-endpoint-beta2
user: beta pass: ****

The downside to moving is you have to update DNS, but once this is done, Varnish will be out of the picture and we'll be able to get a better idea of whether or not it's the problem.

We also considered the possibility that your current SSL endpoint, an Amazon ELB, is bad and needs to be replaced. But if that were the case, many of the requests would be slow or failing, not just a random single slow request. Swapping out the current ELB would also involve a DNS change. Moving to the new SSL Endpoint add-on will make a situation like this easier in the future - if swapping out the ELB is necessary a DNS change would not be needed.

For now, I think moving to SSL Endpoint will be best.

Granted, the latencies I were seeing sometimes got up into the 30-60 SECOND range, which is obviously quite a bit worse than you're seeing. When I run the same apache bench commands against static assets on some of my (HTTPS) cedar apps, I don't do much better (10-15 req/s). When I increase concurrency to 50 or 100, RPS goes up to as much as 30. On Varnish fronted bamboo HTTPS apps, however, I can get it up over 50 req/s for a static asset. If my computer could handle higher concurrency, I suspect it would go even higher. And my internet connection is high latency since it's on a cellular network.