I'm developing a web service that measures site response time. It's a Django app, and as an initial test I pointed it at a couple of my other Django sites on the same VPS. The response time was small (~5ms) most of the time, but on a fairly regular (5 or 10 min) schedule, jumped to a much higher value (up to 400ms, despite no heavy DB load or cache).
Suspicious of my own timing methodology, I pointed it at a static site on the same VPS, and got a consistent quick response. I then used nginx's response_time and upstream_response_time logging to find that it really was my Django apps that were giving the response time variance.
I then pointed the app at a few other Django sites around the web and found similar results: there's a fast "baseline" response with fairly regular spikes to one or two remarkably repeatable slower times. For example, one has a 25ms baseline with ~200ms and ~400ms "steps".
Using the Django Debug Toolbar's Time tab on one of my sites, I can see this behaviour. Most F5 reloads are quick, but there's an occasional (fairly consistently one in ten) slow one, with all the time spent in the "request" section.
Any ideas?
Solved it: it was gunicorn's "max-requests" setting, spawning a new worker every X connections. When X is ten and I'm hitting a "quiet" site, the long response time is every ten hits.
That's why I only noticed the multi-modal times on some Django sites: the others were presumably busy or not using gunicorn.