What is the signal sent to the process running in the container when k8s liveness probe fails? KILL or TERM

Question

I have a use case to gracefully terminate the container where i have a script to kill the process gracefully from within the container by using the command "kill PID".( Which will send the TERM signal ) But I have liveness probe configured as well. Currently liveness probe is configured to probe at 60 second interval. So if the liveness probe take place shortly after the graceful termination signal is sent, the overall health of the container might become CRITICAL when the termination is still in progress. In this case the liveness probe will fail and container will be terminated immediately.

So i wanted to know whether kubelet kills the container with TERM or KILL.

Appreciate your support Thanks in advance

Answer 1

In Kubernetes, Liveness Probe checks for the health state of a container.

To answer your question on whether it uses SIGKILL or SIGTERM, the answer is both are used but in order. So here is what happens under the hood.

1. Liveness probe check fails
2. Kubernetes stops routing of traffic to the container
3. Kubernetes restarts the container
4. Kubernetes starts routing traffic to the container again

For container restart, SIGTERM is first sent with waits for a parameterized grace period, and then Kubernetes sends SIGKILL.

A hack around your issue is to use the attribute:

timeoutSeconds

This specifies how long a request can take to respond before it’s considered a failure. You can add and adjust this parameter if the time taken for your application to come online is predictable.

Also, you can play with readinessProbe before livenessProbe with an adequate delay for the container to come into service after restarting the process. Check https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/ for more details on which parameters to use.