I keep seeing inconsistent performance on a server of mine, and I'm finding it hard to diagnose/fix the problem. For the most part, pages load quickly, but 10-20% of the time they load very slowly. In Chrome's network tab shows "waiting" time as 99% of the lag. I've even tested loading a blank PHP page that has no output and I still see the wait time get large on 10-20% of page views.
I've tried timing the PHP script and it is consistently low on all page views, including the ones that have long wait times. For example, I could load the blank page and it will load in 150ms, and then I load it again and it loads in 8500ms, but both times the PHP script executed in <.5ms.
Any ideas on what else could be causing this (I've tested on multiple computers/networks and am getting reports from users from around the world)?
Connection issues/packet loss
Could it be a high packet loss on your connection? TCP connections sometimes reach timeouts where there is packet loss, whereby it misses a connection initially, but then (in worst case) manages to connect after some seconds worth of retrying.
Of course, you'd notice issues not just viewing your site, but any site on that connection.
Contention on the server
Is it a shared server, and could other clients on the same server be using an excessive amount of IO or CPU, or active Apache processes, at times?
You would debug this with the top command (to see general CPU usage or load queue) or the ps -AH v command (to see the number of active Apache processes and an overview of the other processes running too).