Search code examples
kuberneteskubectl

What happens when you drain nodes in a Kubernetes cluster?

I'd like to get some clarification for preparation for maintenance when you drain nodes in a Kubernetes cluster:

Here's what I know when you run kubectl drain MY_NODE:

  • Node is cordoned
  • Pods are gracefully shut down
  • You can opt to ignore Daemonset pods because if they are shut down, they'll just be re-spawned right away again.

I'm confused as to what happens when a node is drained though.

Questions:

  • What happens to the pods? As far as I know, there's no 'live migration' of pods in Kubernetes.
  • Will the pod be shut down and then automatically started on another node? Or does this depend on my configuration? (i.e. could a pod be shut down via drain and not start up on another node)

I would appreciate some clarification on this and any best practices or advice as well. Thanks in advance.

Solution

  • I just want to add a few things to eamon1234's answer:

    You may find this useful as well:

    1. Link to official docummentation (in case default flags change etc.). According to it:

      The 'drain' evicts or deletes all pods except mirror pods (which cannot be deleted through the API server). If there are DaemonSet-managed pods, drain will not proceed without --ignore-daemonsets, and regardless it will not delete any DaemonSet-managed pods, because those pods would be immediately replaced by the DaemonSet controller, which ignores unschedulable markings. If there are any pods that are neither mirror pods nor managed by ReplicationController, ReplicaSet, DaemonSet, StatefulSet or Job, then drain will not delete any pods unless you use --force. --force will also allow deletion to proceed if the managing resource of one or more pods is missing.

    2. Simple chart illustrating what actually happens when using kubectl drain.

    3. Using kubectl drain with --dry-run option may be also a good idea so you can see its outcome before any actual changes are applied e.g.:

      kubectl drain foo --force --dry-run

      however it will not show any errors about existing local data or daemonsets which you can see whithout using --dry-run flag: ... error: cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore) ...