Search code examples
javanginxsslgoogle-cloud-platformnginx-ingress

Nginx sometimes fails TLS passthrough to Java 17 application


We have a spring-boot application that runs perfectly fine by itself on both Java 11 and Java 17. The spring-boot application is packaged as a docker container and runs inside gcp/gke kubernetes. We use the nginx ingress to forward the traffic with tls-passthrough. We use a Let's Encrypt certificate for our application. The nginx does not have access to it (AFAICT), but considers it valid.

When using Java 11 everything works fine. However, when using Java 17 the first (few) requests pass fine, but then I get a certificate error. The nginx generates/has a default ingress certificate, that it uses for the later requests. But I don't understand why it does serve that (sometimes) in the first place. The error is reproducible with browsers and Java applications. I did not manage to preproduce it with curl/openssl though. After a short time/few minutes the error vanishes for the next (few) requests before it emerges again. When adding the ingress certificate to the trusted certs in browsers I can see that the ingress requests are upgraded to HTTP2, the first few HTTP1 requests all use the correct certificate.

We tried with different java 17 base images (openjdk/eclipse-temurin + alpine/ununtu). We tried to explicitly disable http2 in Java and the browser. Nothing seems to work except for adding the self-signed certificate to the trust-store (which is obviously a no go for production). We weren't able to reproduce this locally, but might be due to our local dev setup being only a simplified version of the cloud environments. If I use kubectl port-forward into the java app container, I cannot reproduce the issue.

We use the following versions:

  • nginx-ingress-1.41.3
  • gke v1.21.6-gke.1500
  • eclipse-temurin 17
  • spring-boot 2.6.3 with the default tomcat

TLDR: The nginx-ingress sometimes does not tls-passthrough to our Java 17 app correctly and thus serves an invalid certificate for those requests. (All responses contain the expected/same/valid content except for the certificate). Has anyone an idea what is happening and how to fix/avoid that?


Solution

  • We had the exact same issue, and we believe the cause to be

    https://github.com/kubernetes/ingress-nginx/issues/9540

    If you cannot upgrade, as stated in the issue

    In the meantime, it seems that setting the Java system property jdk.tls.server.enableSessionTicketExtension to false on the server side will disable the TLS feature which is causing this problem.

    seemed to solve the problem in our case.