Duplicate header "Transfer-Encoding" in response causes client exception - "java.io.IOException: Illegal character in chunk size: 123"

Question

There are 3 Spring Boot services (Client, Proxy, API) - Client is calling API and request is sent to/back through proxy, approx 1% of responses are corrupted and client unable to read it - throwing java.io.IOException: Illegal character in chunk size: 123.

This means that response is chunked and every chunk has size. Since the hex representation of the character "{" is "123" it means that in response chunk size value is missing and data starts with “{“ which is start of JSON object. Client receives header “Transfer-Encoding”: chunked – which is instruction to read the response by chunks, but as the chunk size/ partition size is missing Client unable to read response and throws exception.

Looking into response headers from API, which comes throw Proxy I can see there duplicate header “Transfer-Encoding”. It was found that API itself adding originally such header ONCE, which is normal behavior when response is dynamic (from DB) Spring/Tomcat doesn’t know content length and it chunks the response. But also when response goes through Proxy, then proxy adds one more “Transfer-Encoding” header. Looking in Proxy code – it converts received headers from API and forwards it (all headers are unique, no duplication). But proxy internal Tomcat is adding additionally “Transfer-Encoding” header to response. It looks like Tomcat bug. Upgrading Spring Boot/Tomcat version to the latest did not help.

This problem can be reproduced only during load – e.g 300 requests will give around 3-4 errors. Removing manually “Transfer-Encoding” header (Content-Length will be added automatically) on API side solves the problem, but this doesn’t seem correct solution.

Please share your experience/ suggest a solution.

Proxy has one method inside which it decides where to forward the request. It uses OkHttpClient for that. Here is piece of code which handles requests and converts received headers. (okttp3Headers.names() - is SET, so duplication is not possible)

...         return new ResponseEntity<>(
                responseBodyString,
                convertHeaders(forwarderRequestResponse.headers()),
                Objects.requireNonNull(HttpStatus.resolve(forwarderRequestResponse.code())));
    }


    public HttpHeaders convertHeaders(Headers okttp3Headers) {
        var headers = new HttpHeaders();
        for (String headerName : okttp3Headers.names()) {
            for (String headerValue : okttp3Headers.values(headerName)) {
                headers.add(headerName, headerValue);
            }
        }
        return headers;
    } 

Answer 1

Problem solved by replacing Tomcat with Jetty embedded server. No duplicate headers -> no errors.