Search code examples
spring-bootkuberneteskubernetes-podapplication-lifecyclegraceful-shutdown

How long does it take for Kubernetes to remove a terminating pod from Endpoints?

We are using Spring Boot in Kubernetes in our project. Recently we decided to add graceful shutdown for our application in order to ensure seamless deployments and make sure no requests are failed during pod terminations for whatever reason. Even though Spring Boot provides graceful shutdown, it seems there is still a probability that requests may fail due to the fact that Kubernetes starts removing a pod from endpoints and sends the SIGTERM signal to the pod at the same time in parallel. Quote from kubernetes documentation:

At the same time as the kubelet is starting graceful shutdown, the control plane removes that shutting-down Pod from Endpoints (and, if enabled, EndpointSlice) objects where these represent a Service with a configured selector.

This is aldso described in more detail here. The solution is provided there as well, which is

you can add a sleep in a preStop hook of the pod spec and, of course, configure that sleep to be whatever makes sense for your use case :

enter image description here

In the example provided this sleep is configured to be 10 seconds, but I'm wondering what is the reasonable value for that, so the pod's termination is not unnecessarily delayed?

Thanks.

Solution

  • It seems like this might be up to your preferences but it looks like the 5-10 seconds sleep is a recommended range:

    In “Kubernetes in Action”, Lukša recommends 5–10 seconds https://blog.gruntwork.io/delaying-shutdown-to-wait-for-pod-deletion-propagation-445f779a8304