Search code examples
spring-mvctomcattomcat7http-status-code-403

Unusual 403 behaviour on Spring MVC / Tomcat with many simultaneous requests

I have a Spring 4.2.3 MVC application with Hibernate, Jackson, and a long list of other libraries that is exhibiting strange behaviour on one server only, with four or five other near-identical servers behaving normally. The application runs in Tomcat 7 on Java 8.

A @RestController controller uses @PreAuthorize to authorize each request to an endpoint. When a user is missing the required permission that request returns a 403 (Forbidden) response, invalidates the session, and all following requests return a 401 (Unauthorized) until that user authenticates again.

The problem server spontaneously returns 403s when the user has all required permissions. Our permission evaluator logs when a user is missing a permission and in this case it doesn't log anything (including no exceptions). The server returns multiple 403 responses, not just one followed by 401s. Additionally the 403 response is the default Apache / Tomcat output whereas we have Spring configured to return a basic HTML page with its 403 for legacy scenarios where the user will see it. All this makes me think that the issue exists outside of the Spring application, perhaps in Tomcat.

In certain situations the client can send a large number of HTTP requests to the same endpoint - up to 42. I have seen the first 30 or so return 200 OK and the following 12 all return 403. The only way I've been able to suppress the spontaneous 403s is to consolidate the 42 into four HTTP requests. This consolidation is a good thing regardless of the problem, but I'd like to understand the mechanism at work here. If I don't know the limit of how many requests the server can handle then I'll never be able to trust it.

My thoughts so far are:

  1. some built-in DOS protection decides that I have too many simultaneous requests and responds in the least resource-intensive way
  2. Tomcat hits a thread limit and reacts in an undesirable way

What could be causing this behaviour on one server out of several?

Solution

  • The problem was Apache's mod_evasive module. The problem server was configured with DoS / DDoS protection because it was available externally. mod_evasive exists to provide this exact behaviour - if Apache receives too many requests based on its configuration it returns 403 responses as a low-resource shortcut.