I am using Helm Charts for deployments on google-kubernetes-engine and using rolling-update in it.
Currently I am running 10 pods. When I make deployment using rolling-update, I expect a new pod comes up and traffic is stopped from the old pod going down and then it is gracefully taken down. And so on for next pods.
But in my case when a new pod is created, old pod immediately goes down and I start getting Internal Server Error [500] for requests being fulfilled by that pod.
How can I avoid this?
livenessProbe:
httpGet:
path: /health
port: 4000
initialDelaySeconds: 1
periodSeconds: 10
readinessProbe:
httpGet:
path: /health
port: 4000
initialDelaySeconds: 1
periodSeconds: 10
It sounds like you need to tweak your rolling update strategy. You can find similar discussions here and here concerning performing rolling updates without getting errors.
The upgrade strategy is important to define how many unavailable pods you can have during the update. For now downtime, you may want to set this to 0 and configure a reasonable maxSurge value.
The next step is to make sure that you have appropriate readinessProbes configured. Once the new pod is marked as ready, the controller will attempt to remove one (or more) of the old pods. Your pod will receive a SIGTERM and proceed to handle that however it is configured to do so. This means:
A) Make sure the readinessProbe only marks a pod as ready once it is fully able to accept traffic (/health may be up even though the application is not, make sure this is not the case).
B) Your old pods need to handle SIGTERM properly and gracefully, this is done at the application layer. Keep in mind that by default, the controller will allow the pods to shut down gracefully once the SIGTERM is sent.