We have a service that needs to run a few longer SQL queries when it shuts down. However, when the pod receives a SIGTERM from Kubernetes, the istio proxy container waits only 5s prior to shutting down. This causes our queries to fail and the service terminates ungracefully.
Things we've tried:
terminationGracePeriodSeconds to 3600. Istio still shuts down after 5s.TERMINATION_DRAIN_DURATION_SECONDS to 3600 on the istio container. Istio keeps running until 3600s have elapsed, even if our service has finished shutting down. We tried calling curl -XPOST http://127.0.0.1:15000/quitquitquit to get Istio to shut down sooner but it remains running for the full time.How can we get Istio to stay running long enough for our service to terminate gracefully, without having it stay running for too long?
As far as I know unfortunately Graceful shutdown for istio sidecar is not supported:
Currently there is not. This seems like an interesting feature request though, it may be worth a feature request on github.
I also think that setting up an appropriate thread on github will be a solution.
The only way I know is to use TERMINATION_DRAIN_DURATION_SECONDS option.
There are also several topic on github related to this theme:
istio-proxy stop before my containers #10112
And currently envoy does not support graceful shutdown. xref: envoyproxy/envoy#2920 Once envoy implements this, we can support it in sidecar.
For anyone looking through issues for why their connection to a pod is disconnecting early, hopefully this helps.
You need the pod to have a
terminationGracePeriodSeconds, which I assume you were already aware about, and you also need an annotation for the pod's sidecar config:annotations: # https://istio.io/latest/docs/reference/config/istio.mesh.v1alpha1/#ProxyConfig proxy.istio.io/config: | terminationDrainDuration: {{ $terminationGracePeriodSeconds }}s
Envoy shutting down before the thing it's wrapping can cause failed requests #7136