Search code examples
kuberneteskubernetes-pod

K8s - Schedule new pod before the old one is terminated


I have read up on the Kubernetes docs but I'm unable to get a clear answer on my question. I'm using the official cluster-autoscaler.

  1. I have a deployment that specifies one replica should be running. When a pod is terminated (for example, was running on a node that is getting scaled-down) is the new pod scheduled before the termination begins or after the termination is done? The docs say that schedule happens when terminating, but don't mention at which phase.
  2. To achieve seamless node scale-down without disruption to any services, I would expect k8 to scale up pods to N+1 replicas (at this point pods are scheduled only to nodes that are not scaling down) and then drain the node. Based on my testing, it first drains, and then schedules any missing pods based on configurations. Is it possible to configure this behaviour or this is currently not possible to do?

From what I understand, seamless updates are easy with RollingUpdate strategy. I have not find the same "Rolling" strategy to be possible for scale-down.

EDIT

TL;DR I'm looking for HA on a) two+ replica deployment and b) one replica deployment

a) Can be achieved by using PDBs. Checkout Fritz's answer. If you need pods scheduled on different nodes, leverage anti-affinity (Marc's answer)

b) If you're okay with short disruption, PDB is the official way to go. If you need a workaround, my answer can be of inspiration.


Solution

  • The scale down behavior can be configured with what is called a Disruption Budget

    In your Deployment Manifest you can define maxUnavailable and minAvailable number of Pods during voluntary disruptions like draining nodes.

    For how to do it, check out the K8s Documentation.