we are upgrading from Spring Boot version 3.2.5
Some background info, we have a microservice Client Service API that is an API gateway & forwards the requests to the corresponding microservice
In this case, the Client Service API microservice checks the status of a microservice by checking its /check-status endpoint.
When using Spring Boot starter web 3.4.1, we noticed the following behavior:
[Client Service API] -- DEBUG [https-jsse-nio-8200-exec-8] - [2025-01-13 10:15:43,251] - POST "/api/callService/check-status/ms1", parameters={}
[Client Service API] -- DEBUG [https-jsse-nio-8200-exec-8] - [2025-01-13 10:15:43,251] - Mapped to com.co.team.clapi.rest.ClientAPIController#healthCheck(String, Object, HttpServletRequest)
[Client Service API] -- DEBUG [https-jsse-nio-8200-exec-8] - [2025-01-13 10:15:43,252] - Inside getInputStream overrriden method
[Client Service API] -- DEBUG [https-jsse-nio-8200-exec-8] - [2025-01-13 10:15:43,252] - Read "application/json;charset=UTF-8" to [{serviceName=null, status=null, message=null}]
[Client Service API] -- INFO [https-jsse-nio-8200-exec-8] - [2025-01-13 10:15:43,253] - Health check request received for svcName: bsa
[Client Service API] -- INFO [https-jsse-nio-8200-exec-8] - [2025-01-13 10:15:43,253] - Header name: accept Header Value: text/plain, application/json, application/*+json, */*
[Client Service API] -- INFO [https-jsse-nio-8200-exec-8] - [2025-01-13 10:15:43,253] - Header name: content-type Header Value: application/json
[Client Service API] -- INFO [https-jsse-nio-8200-exec-8] - [2025-01-13 10:15:43,253] - Header name: authorization Header Value: Bearer~eyJhbGciOiJIUzUxMiJ9.eyJzdWIiOiJDT1JFQ05GVCIsImF1ZCI6WyIxMC4yMTEuNTEuMTA3Il0sInZlciI6MSwiY3JlYXRlZCI6MTczNjc2MzM0MzIzOCwiaXNzIjoiMTAuMjExLjUxLjEwNyIsImp0aSI6IkNPUkVDTkZUdFFySEFOVkdpTVhXVSIsImV4cCI6MTczNjc2NDI0M30.6D4l3Df4hMApfcavOc_jTpuwRLw-t2SQX1oo4F2jsux_fdRYvK7o5k0_J4QNbSarAILePtTZToTozBNL82h_gg
[Client Service API] -- INFO [https-jsse-nio-8200-exec-8] - [2025-01-13 10:15:43,253] - Header name: accept-encoding Header Value: gzip, x-gzip, deflate
[Client Service API] -- INFO [https-jsse-nio-8200-exec-8] - [2025-01-13 10:15:43,253] - Header name: transfer-encoding Header Value: chunked
[Client Service API] -- INFO [https-jsse-nio-8200-exec-8] - [2025-01-13 10:15:43,253] - Header name: host Header Value: client-dev.trusted.com
[Client Service API] -- INFO [https-jsse-nio-8200-exec-8] - [2025-01-13 10:15:43,253] - Header name: connection Header Value: keep-alive
[Client Service API] -- INFO [https-jsse-nio-8200-exec-8] - [2025-01-13 10:15:43,253] - Header name: user-agent Header Value: Apache-HttpClient/5.3.1 (Java/17.0.12)
[Client Service API] -- INFO [https-jsse-nio-8200-exec-8] - [2025-01-13 10:15:43,253] - Header name: x-forwarded-for Header Value: 10.211.51.107
[Client Service API] -- INFO [https-jsse-nio-8200-exec-8] - [2025-01-13 10:15:43,255] - Request will be forwarded to: https://microservice-dev.trusted.com/ms1/check-status
[Client Service API] -- DEBUG [https-jsse-nio-8200-exec-8] - [2025-01-13 10:15:43,255] - HTTP POST https://microservice-dev.trusted.com/ms1/check-status
[Client Service API] -- DEBUG [https-jsse-nio-8200-exec-8] - [2025-01-13 10:15:43,256] - Accept=[text/plain, application/json, application/*+json, */*]
[Client Service API] -- DEBUG [https-jsse-nio-8200-exec-8] - [2025-01-13 10:15:43,256] - Writing [{serviceName=null, status=null, message=null}] with org.springframework.http.converter.json.MappingJackson2HttpMessageConverter
[Client Service API] -- DEBUG [https-jsse-nio-8200-exec-8] - [2025-01-13 10:15:43,272] - Looks like the client has gone away: org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://microservice-dev.trusted.com/ms1/check-status": Connection reset (For a full stack trace, set the log category 'org.apache.commons.logging.LogAdapter$Log4jLog@2de1467f' to TRACE level.)
[Client Service API] -- DEBUG [https-jsse-nio-8200-exec-8] - [2025-01-13 10:15:43,272] - Completed 200 OK
In this case, the microservice being checked is not active & Client Service API should return a 500 HTTP status code, instead of 200.
I see this line in the log, what's causing this?
Looks like the client has gone away: org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://microservice-dev.trusted.com/ms1/check-status": Connection reset (For a full stack trace, set the log category 'org.apache.commons.logging.LogAdapter$Log4jLog@2de1467f' to TRACE level.)
Refer below the current log when using version 3.2.7 of Spring Boot starter web:
[Client Service API] -- DEBUG [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,759] - POST "/api/callService/check-status/ms1", parameters={}
[Client Service API] -- DEBUG [https-jsse-nio-8200-exec-8] - [2025-01-13 10:15:43,251] - Mapped to com.co.team.clapi.rest.ClientAPIController#healthCheck(String, Object, HttpServletRequest)
[Client Service API] -- DEBUG [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,760] - Inside getInputStream overrriden method
[Client Service API] -- DEBUG [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,760] - Read "application/json;charset=UTF-8" to [{serviceName=null, status=null, message=null}]
[Client Service API] -- INFO [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,760] - Health check request received for svcName: ms1
[Client Service API] -- INFO [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,760] - Header name: accept Header Value: text/plain, application/json, application/*+json, */*
[Client Service API] -- INFO [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,760] - Header name: content-type Header Value: application/json
[Client Service API] -- INFO [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,760] - Header name: authorization Header Value: Bearer~eyJhbGciOiJIUzUxMiJ9.eyJzdWIiOiJDT1JFQ05GVCIsImF1ZCI6WyIxMC4yMTEuNTEuMTA3Il0sInZlciI6MSwiY3JlYXRlZCI6MTczNjkyMzMyNzc0OSwiaXNzIjoiMTAuMjExLjUxLjEwNyIsImp0aSI6IkNPUkVDTkZURkZEZU9pR2RBMThmSCIsImV4cCI6MTczNjkyNDIyN30.2uQsDJxwK3MYoPW_I_DV9VfwwWKe4qWXtcEhfC8w0EPXgucAdcneIn2Mq0fEw1ROiWjt7AZfZrW7sYOs12Lcqw
[Client Service API] -- INFO [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,760] - Header name: accept-encoding Header Value: gzip, x-gzip, deflate
[Client Service API] -- INFO [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,760] - Header name: transfer-encoding Header Value: chunked
[Client Service API] -- INFO [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,760] - Header name: host Header Value: client-dev.trusted.com
[Client Service API] -- INFO [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,760] - Header name: connection Header Value: keep-alive
[Client Service API] -- INFO [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,760] - Header name: user-agent Header Value: Apache-HttpClient/5.3.1 (Java/17.0.12)
[Client Service API] -- INFO [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,760] - Header name: x-forwarded-for Header Value: 10.211.51.107
[Client Service API] -- INFO [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,762] - Request will be forwarded to: https://microservice-dev.trusted.com/ms1/check-status
[Client Service API] -- DEBUG [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,762] - HTTP POST https://microservice-dev.trusted.com/ms1/check-status
[Client Service API] -- DEBUG [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,762] - Accept=[text/plain, application/json, application/*+json, */*]
[Client Service API] -- DEBUG [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,762] - Writing [{serviceName=null, status=null, message=null}] with org.springframework.http.converter.json.MappingJackson2HttpMessageConverter
[Client Service API] -- DEBUG [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,774] - Failed to complete request: org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://microservice-dev.trusted.com/ms1/check-status": Connection reset
[Client Service API] -- DEBUG [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,775] - Securing POST /error
[Client Service API] -- DEBUG [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,775] - Set SecurityContextHolder to anonymous SecurityContext
[Client Service API] -- DEBUG [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,775] - Authorized filter invocation [POST /error] with attributes [permitAll]
[Client Service API] -- DEBUG [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,775] - Secured POST /error
[Client Service API] -- DEBUG [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,775] - "ERROR" dispatch for POST "/error", parameters={}
[Client Service API] -- DEBUG [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,775] - Mapped to org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController#error(HttpServletRequest)
[Client Service API] -- DEBUG [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,776] - Using 'application/json', given [text/plain, application/json, application/*+json, */*] and supported [application/json, application/*+json]
[Client Service API] -- DEBUG [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,776] - Writing [{timestamp=Wed Jan 15 06:42:07 GMT 2025, status=500, error=Internal Server Error, path=/api/callServ (truncated)...]
[Client Service API] -- DEBUG [https-jsse-nio-8200-exec-9] - [2025-01-15 06:42:07,776] - Exiting from "ERROR" dispatch, status 500
I checked the 3.4.1 changelog here & don't see any specific mention of changes that could be causing this? link.
if ("true".equalsIgnoreCase(sslEnabled.trim())) {
logger.info("Request will be forwarded to: " + "https://" + serviceUrl);
return restTemplate.postForObject("https://" + serviceUrl, reqObj, String.class);
}
logger.info("Request will be forwarded to: " + "http://" + serviceUrl);
return restTemplate.postForObject("http://" + serviceUrl, reqObj, String.class);
Can anyone assist? I am not sure how to navigate this.
The message "Looks like the client has gone away" comes from Spring Web's DisconnectedClientHelper. This class analyzes the type and message of the exception that happened during the processing of the request. It tries to detect if the error happened because of the client (e.g., it didn't bother to wait until the response is received and reset the connection). This is needed to avoid polluting the logs with client disconnect exceptions.
Starting from Spring Web version 6.10.0, this class started treating the exception message "Connection reset", among others, as a sign that the client has gone away. But in your case, this message comes not from the client, but from the downstream service you're trying to access, because it's not yet ready.
Starting from Spring Boot 3.4.0, this case also causes the exception to be ignored by the framework. This might mean a bug / misconception in the Spring Web framework (it assumes the exception "Connection reset" is always caused by the client, whereas it may be caused by the downstream server). Your controller method still throws an exception, it's just being silently ignored by the framework.
To fix the issue on your side, you can add an exception handler to your controller, similar to this:
@ExceptionHandler(RestClientException.class)
public ResponseEntity<String> handle(RestClientException e) {
return ResponseEntity.internalServerError().body(e.getMessage());
}
This handler will handle the exception that your method throws and return a proper response.