Hello Stack Overflow Community,
Been a while since I've asked a question so if I forget some information, just let me know and I will gladly add.
We have recently uncovered some issues occurring in our client-server. We are running a Spring server that handles client requests and renders HTML via Thymeleaf. We are using Zuul as a way to redirect our many client routes to a single Spring endpoint that serves up a single bundle so that no matter where a user refreshes, they will receive the bundle we want them to get. This seems to work most of the time but once every minute or so (out of thousands a minute) we get a zuul forwarding exception like so:
[2m2020-11-30 02:26:51.023[0;39m [33m WARN[0;39m [35m50[0;39m [2m---[0;39m [33m[trace=,span=][0;39m [2m[nio-8080-exec-2][0;39m [36mo.s.c.n.z.filters.post.SendErrorFilter [0;39m
[2m:[0;39m Error during filteringcom.netflix.zuul.exception.ZuulException: Forwarding error
at
org.springframework.cloud.netflix.zuul.filters.route.SimpleHostRoutingFilter.handleException(SimpleHostRoutingFilter.java:261) ~[spring-cloud-netflix-zuul-2.2.1.RELEASE.jar:2.2.1.RELEASE]
at
org.springframework.cloud.netflix.zuul.filters.route.SimpleHostRoutingFilter.run(SimpleHostRoutingFilter.java:241) ~[spring-cloud-netflix-zuul-2.2.1.RELEASE.jar:2.2.1.RELEASE]
at
com.netflix.zuul.ZuulFilter.runFilter(ZuulFilter.java:117) ~[zuul-core-1.3.1.jar:1.3.1]
at
com.netflix.zuul.FilterProcessor.processZuulFilter(FilterProcessor.java:193) ~[zuul-core-1.3.1.jar:1.3.1]
at com.netflix.zuul.FilterProcessor.runFilters(FilterProcessor.java:157) ~[zuul-core-1.3.1.jar:1.3.1]
at
com.netflix.zuul.FilterProcessor.route(FilterProcessor.java:118) ~[zuul-core-1.3.1.jar:1.3.1]
at
com.netflix.zuul.ZuulRunner.route(ZuulRunner.java:96) ~[zuul-core-1.3.1.jar:1.3.1]
at
com.netflix.zuul.http.ZuulServlet.route(ZuulServlet.java:116) ~[zuul-core-1.3.1.jar:1.3.1]
at
com.netflix.zuul.http.ZuulServlet.service(ZuulServlet.java:81) ~[zuul-core-1.3.1.jar:1.3.1]
at
org.springframework.web.servlet.mvc.ServletWrappingController.handleRequestInternal(ServletWrappingController.java:166) [spring-webmvc-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at
org.springframework.cloud.netflix.zuul.web.ZuulController.handleRequest(ZuulController.java:45) [spring-cloud-netflix-zuul-2.2.1.RELEASE.jar:2.2.1.RELEASE]
at
org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:52) [spring-webmvc-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) [spring-webmvc-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) [spring-webmvc-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) [spring-webmvc-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) [spring-webmvc-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:626) [tomcat-embed-core-9.0.37.jar:4.0.FR]
at
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) [spring-webmvc-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:733) [tomcat-embed-core-9.0.37.jar:4.0.FR]
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) [tomcat-embed-websocket-9.0.37.jar:9.0.37]
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.springframework.web.servlet.resource.ResourceUrlEncodingFilter.doFilter(ResourceUrlEncodingFilter.java:64) [spring-webmvc-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) [spring-web-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) [spring-web-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:109) [spring-boot-actuator-2.2.7.RELEASE.jar:2.2.7.RELEASE]
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) [spring-web-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.cloudfoundry.router.ClientCertificateMapper.doFilter(ClientCertificateMapper.java:79) [client_certificate_mapper-1.11.0_RELEASE.jar:na]
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1589) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.37.jar:9.0.37]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_242]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_242]
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.37.jar:9.0.37]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_242]
Caused by: org.apache.http.NoHttpResponseException: The target server failed to respond
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141) ~[httpclient-4.5.12.jar:4.5.12]
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[httpclient-4.5.12.jar:4.5.12]
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[httpcore-4.4.13.jar:4.4.13]
at
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[httpcore-4.4.13.jar:4.4.13]
at
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) ~[httpclient-4.5.12.jar:4.5.12]
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.13.jar:4.4.13]
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.13.jar:4.4.13]
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[httpclient-4.5.12.jar:4.5.12]
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.12.jar:4.5.12]
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.12.jar:4.5.12]
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.12.jar:4.5.12]
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118) ~[httpclient-4.5.12.jar:4.5.12]
at
org.springframework.cloud.netflix.zuul.filters.route.SimpleHostRoutingFilter.forwardRequest(SimpleHostRoutingFilter.java:422) ~[spring-cloud-netflix-zuul-2.2.1.RELEASE.jar:2.2.1.RELEASE]
at
org.springframework.cloud.netflix.zuul.filters.route.SimpleHostRoutingFilter.forward(SimpleHostRoutingFilter.java:341) ~[spring-cloud-netflix-zuul-2.2.1.RELEASE.jar:2.2.1.RELEASE]
at
org.springframework.cloud.netflix.zuul.filters.route.SimpleHostRoutingFilter.run(SimpleHostRoutingFilter.java:236) ~[spring-cloud-netflix-zuul-2.2.1.RELEASE.jar:2.2.1.RELEASE]
... 61 common frames omitted
As you can see, we are using HttpClient 4.5.12. I saw similar issues occurring in 4.4 in this post but we upgraded and the issue persisted. In the same post @daimarom commented that perhaps the PoolingHttpClientConnectionManager could be causing an issue. This is how we configured our pool manager:
@Bean
public PoolingHttpClientConnectionManager connectionManager() {
PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
connectionManager.setValidateAfterInactivity(5000);
connectionManager.setDefaultMaxPerRoute(40);
connectionManager.setMaxTotal(15);
return connectionManager;
}
@Bean
public CloseableHttpClient httpClient() throws Exception {
return HttpClients.custom()
.disableAutomaticRetries()
.setSSLContext(sslContext())
.setSSLSocketFactory(socketFactory())
.setConnectionManager(connectionManager())
.setSSLHostnameVerifier(new NoopHostnameVerifier())
.build();
}
Playing aroung with the connection manager parameters did not help.
We had a Zuul filter but made our routing completely configuration based to rule out any logical errors. Currently our Zuul configuration looks like such:
zuul:
sensitive-headers:
host.socket-timeout-millis: 60000
routes:
api:
strip-prefix: false
path: /api/**
url: "${validation.urls.piserver}"
extract:
strip-prefix: true
path: /web2/extract/**
url: "${validation.urls.piclient}/validate"
legal:
strip-prefix: true
path: /web2/legal/**
url: "${validation.urls.piclient}/validate"
capture-vin:
strip-prefix: true
path: /web2/capture-instructions/vin/**
url: "${validation.urls.piclient}/validate"
recapture-vin:
strip-prefix: true
path: /web2/recapture-instructions/vin/**
url: "${validation.urls.piclient}/validate"
capture-odometer:
strip-prefix: true
path: /web2/capture-instructions/odometer/**
url: "${validation.urls.piclient}/validate"
recapture-odometer:
strip-prefix: true
path: /web2/recapture-instructions/odometer/**
url: "${validation.urls.piclient}/validate"
capture-rear:
strip-prefix: true
path: /web2/capture-instructions/rear/**
url: "${validation.urls.piclient}/validate"
recapture-rear:
strip-prefix: true
path: /web2/recapture-instructions/rear/**
url: "${validation.urls.piclient}/validate"
capture-rear-passenger:
strip-prefix: true
path: /web2/capture-instructions/rear-passenger/**
url: "${validation.urls.piclient}/validate"
recapture-rear-passenger:
strip-prefix: true
path: /web2/recapture-instructions/rear-passenger/**
url: "${validation.urls.piclient}/validate"
capture-front-passenger:
strip-prefix: true
path: /web2/capture-instructions/front-passenger/**
url: "${validation.urls.piclient}/validate"
recapture-front-passenger:
strip-prefix: true
path: /web2/recapture-instructions/front-passenger/**
url: "${validation.urls.piclient}/validate"
capture-front-driver:
strip-prefix: true
path: /web2/capture-instructions/front-driver/**
url: "${validation.urls.piclient}/validate"
recapture-front-driver:
strip-prefix: true
path: /web2/recapture-instructions/front-driver/**
url: "${validation.urls.piclient}/validate"
capture-rear-driver:
strip-prefix: true
path: /web2/capture-instructions/rear-driver/**
url: "${validation.urls.piclient}/validate"
recapture-rear-driver:
strip-prefix: true
path: /web2/recapture-instructions/rear-driver/**
url: "${validation.urls.piclient}/validate"
capture-damage-left:
strip-prefix: true
path: /web2/capture-instructions/damage-left/**
url: "${validation.urls.piclient}/validate"
recapture-damage-left:
strip-prefix: true
path: /web2/recapture-instructions/damage-left/**
url: "${validation.urls.piclient}/validate"
capture-damage-center:
strip-prefix: true
path: /web2/damage-capture-instructions/damage-center/**
url: "${validation.urls.piclient}/validate"
recapture-damage-center:
strip-prefix: true
path: /web2/damage-recapture-instructions/damage-center/**
url: "${validation.urls.piclient}/validate"
capture-damage-right:
strip-prefix: true
path: /web2/damage-capture-instructions/damage-right/**
url: "${validation.urls.piclient}/validate"
recapture-damage-right:
strip-prefix: true
path: /web2/damage-recapture-instructions/damage-right/**
url: "${validation.urls.piclient}/validate"
capture-damages:
strip-prefix: true
path: /web2/capture-instructions/damages/**
url: "${validation.urls.piclient}/validate"
recapture-damages:
strip-prefix: true
path: /web2/recapture-instructions/damages/**
url: "${validation.urls.piclient}/validate"
uploading:
strip-prefix: true
path: /web2/uploading/**
url: "${validation.urls.piclient}/validate"
submit:
strip-prefix: true
path: /submit/**
url: "${validation.urls.piclient}/validate"
intake-processing:
strip-prefix: true
path: /intake-processing/**
url: "${validation.urls.piclient}/validate"
already-submitted:
strip-prefix: true
path: /already-submitted/**
url: "${validation.urls.piclient}/validate"
submit-v1:
strip-prefix: true
path: /submit
url: "${validation.urls.piclient}/validate"
intake-processing-v1:
strip-prefix: true
path: /intake-processing
url: "${validation.urls.piclient}/validate"
already-submitted-v1:
strip-prefix: true
path: /already-submitted
url: "${validation.urls.piclient}/validate"
This used to be much more succinct in the routing filter when we simply reset the request URI in the current context but as I mentioned before, now this is all configuration based.
All of our dependencies are defined here:
dependencies {
compileOnly 'org.projectlombok:lombok'
annotationProcessor 'org.projectlombok:lombok'
annotationProcessor 'org.springframework.boot:spring-boot-configuration-processor'
implementation 'org.yaml:snakeyaml:1.26'
implementation 'io.github.openfeign:feign-httpclient:10.7.4'
implementation 'org.springframework.boot:spring-boot-starter-web'
implementation 'org.springframework.boot:spring-boot-starter-thymeleaf'
implementation 'org.springframework.boot:spring-boot-starter-actuator'
implementation 'org.springframework.cloud:spring-cloud-starter-openfeign'
implementation 'org.springframework.cloud:spring-cloud-starter-netflix-zuul'
implementation 'org.apache.tomcat.embed:tomcat-embed-core:9.0.37'
implementation 'org.apache.tomcat.embed:tomcat-embed-el:9.0.37'
implementation 'org.apache.tomcat.embed:tomcat-embed-websocket:9.0.37'
implementation 'org.springframework.cloud:spring-cloud-cloudfoundry-connector'
testImplementation 'org.springframework.security:spring-security-test'
testImplementation('org.springframework.boot:spring-boot-starter-test') {
exclude group: 'org.junit.vintage', module: 'junit-vintage-engine'
}
}
I feel like I am missing something completely obvious here but our manager who was very experienced with Zuul recently left. Please let me know if there is any additional information I can add here. Thank you in advance for any help.
[UPDATE] I have recently attempted taking the entire pool manager out of the configuration. The problem has persisted. Now the configuration for the HttpClient looks like the following:
@Bean
public CloseableHttpClient httpClient() throws Exception {
return HttpClients.custom()
.disableAutomaticRetries()
.setSSLContext(sslContext())
.setSSLSocketFactory(socketFactory())
// .setConnectionManager(connectionManager())
.setSSLHostnameVerifier(new NoopHostnameVerifier())
.build();
}
Also, we tried with and without a spring security configuration chain explicitly defined. Nothing helped.
Ok so @brijesh mentioned a potential solution in the comments that I have yet to try but I did end up deploying another "fixing" change. It's more of a recovery from the issue which is mostly likely still occurring. I simply enabled automatic retrying by commenting out the following line:
@Bean
public CloseableHttpClient httpClient() throws Exception {
return HttpClients.custom()
// .disableAutomaticRetries()
.setSSLContext(sslContext())
.setSSLSocketFactory(socketFactory())
.setConnectionManager(connectionManager())
.setSSLHostnameVerifier(new NoopHostnameVerifier())
.build();
}
For anyone who does not explicitly configure their HttpClient settings, this shouldn't be an issue since the default is to retry 3 times.
For people who do configure these settings and are having issues with the connection manager, I will soon attempt @brijesh's approach (with automatic retrying disabled) to see if the solution resolves the root cause. I will update the answer here once I've completed this.