Search code examples
apachetomcatajp

Apache/Tomcat error - wrong pages being delivered

This error has been driving me nuts. We have a server running Apache and Tomcat, serving multiple different sites. Normally the server runs fine, but sometimes an error happens where people are served the wrong page - the page that somebody else requested!

Clues:

  • The pages being delivered are those that another user requested recently, and are otherwise delivered correctly. It's been known for two simultaneous requests to be swapped. As far as I can tell, none of the pages being incorrectly delivered are older than a few minutes.
  • It only affects the files that are being served by Tomcat. Static files like images are unaffected.
  • It doesn't happen all the time. When it does happen, it happens for everybody.
  • It seems to happen at times of peak demand. However, the demand is not yet very high - it's certainly well within the bounds of what Apache can cope with.
  • Restarting Tomcat fixed it, but only for a few minutes. Restarting Apache fixed it, but only for a few minutes.
  • The server is running Apache 2 and Tomcat 6, using a Java 6 VM on Gentoo. The connection is with AJP13, and JkMount directives within <VirtualHost> blocks are correct.
  • There's nothing of use in any of the log files.

Further information:

Apache does not have any form of caching turned on. All the caching-related entries in httpd.conf and related imports say, for example:

<IfDefine CACHE>
  LoadModule cache_module modules/mod_cache.so
</IfDefine>

While the options for Apache don't include that flag:

APACHE2_OPTS="-D DEFAULT_VHOST -D INFO -D LANGUAGE -D SSL -D SSL_DEFAULT_VHOST -D PHP5 -D JK"

Tomcat likewise has no caching options switched on, that I can find.

toolkit's suggestion was good, but not appropriate in this case. What leads me to believe that the error can't be within my own code is that it isn't simply a few values that are being transferred - it's the entire request, including the URL, parameters, session cookies, the whole thing. People are getting pages back saying "You are logged in as John", when they clearly aren't.

Update:

Based on suggestions from several people, I'm going to add the following HTTP headers to Tomcat-served pages to disable all forms of caching:

Cache-Control: no-store
Vary: *

Hopefully these headers will be respected not just by Apache, but also by any other caches or proxies that may be in the way. Unfortunately I have no way of deliberately reproducing this error, so I'm just going to have to wait and see if it turns up again.

I notice that the following headers are being included - could they be related in any way?

Connection: Keep-Alive
Keep-Alive: timeout=5, max=66

Update:

Apparently this happened again while I was asleep, but has stopped happening now I'm awake to see it. Again, there's nothing useful in the logs that I can see, so I have no clues to what was actually happening or how to prevent it.

Is there any extra information I can put in Apache or Tomcat's logs to make this easier to diagnose?

Update:

Since this has happened again a couple of times, we've changed how Apache connects to Tomcat to see if it affects things. We were using mod_jk with a directive like this:

JkMount /portal ajp13

We've switched now to using mod_proxy_ajp, like so:

ProxyPass /portal ajp://localhost:8009/portal

We'll see if it makes any difference. This error was always annoyingly unpredictable, so we can never definitively say if it's worked or not.

Update:

We just got the error briefly on a site that was left using mod_jk, while a sister site on the same server using mod_proxy_ajp didn't show the error. This doesn't prove anything, but it does provide evidence that swithing to mod_proxy_ajp may have helped.

Update:

We just got the error again last night on a site using mod_proxy_ajp, so clearly that hasn't solved it - mod_jk wasn't the source of the problem. I'm going to try the anonymous suggestion of turning off persistent connections:

KeepAlive Off

If that fails as well, I'm going to be desperate enough to start investigating GlassFish.

Update:

Dammit! The problem just came back. I hadn't seen it in a while, so I was starting to think we'd finally sorted it. I hate heisenbugs.

Solution

  • We switched Apache from proxying with AJP to proxying with HTTP. So far it appears to have solved the issue, or at least vastly reduced it - the problem hasn't been reported in months, and the app's use has increased since then.

    The change is in Apache's httpd.conf. Having started with mod_jk:

    JkMount /portal ajp13

    We switched to mod_proxy_ajp:

    ProxyPass /portal ajp://localhost:8009/portal

    Then finally to straight mod_proxy:

    ProxyPass /portal http://localhost:8080/portal

    You'll need to make sure Tomcat is set up to serve HTTP on port 8080. And remember that if you're serving /, you need to include / on both sides of the proxy or it starts crying:

    ProxyPass / http://localhost:8080/