Running on Kubernetes v1.20
I have a startup probe configured and a liveness probe. On the first start of the container, the startup probe is executed until the liveness probe takes over (as documented). However, it seems if the liveness probe fails and the container restarts, the startup probe is not executed again. Is this intended behavior? I cannot find this documented anywhere.
To reproduce this issue, I'm running the following container definition (relevant parts only):
containers:
- args:
- /bin/sh
- -c
- touch /tmp/alive; sleep 10000
image: busybox
livenessProbe:
exec:
command:
- /bin/sh
- -c
- touch /tmp/liveness; test -f /tmp/alive
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 2
successThreshold: 1
timeoutSeconds: 2
startupProbe:
exec:
command:
- touch
- /tmp/startup
failureThreshold: 3
periodSeconds: 2
successThreshold: 1
timeoutSeconds: 2
So if the liveness probe runs, it creates /tmp/liveness. If the startup probe runs, it creates /tmp/startup. You can simulate the liveness check failing by deleting /tmp/alive.
On first startup:
$ ls /tmp/
alive liveness startup
After rm /tmp/alive, the liveness check fails and the container is restarted. Then, in the new container:
$ ls /tmp/
alive liveness
So it seems the startup probe is not executed anymore.
Might want to check if your K8 version is the one affected by this issue:
https://github.com/kubernetes/kubernetes/issues/101064
or