How to gracefully shutdown a Go service running on Kubernetes

Question

I have an API written in Go that has been Dockerised and runs in a Kubernetes cluster on GKE.

At the moment my API server does not handle any shutdown scenarios such as a Pod dying or being purposefully brought down.

What set of UNIX signals should I expect to trap to gracefully shutdown the server and what circumstances would trigger them? For example, crashes, K8s shutdowns etc.

Answer 1

Kubernetes sends a SIGTERM signal. So the graceful shutdown may look like this:

package main

import (
    "context"
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"
)

func main() {
    var srv http.Server

    idleConnsClosed := make(chan struct{})
    go func() {
        sigint := make(chan os.Signal, 1)

        // interrupt signal sent from terminal
        signal.Notify(sigint, os.Interrupt)
        // sigterm signal sent from kubernetes
        signal.Notify(sigint, syscall.SIGTERM)

        <-sigint

        // We received an interrupt signal, shut down.
        if err := srv.Shutdown(context.Background()); err != nil {
            // Error from closing listeners, or context timeout:
            log.Printf("HTTP server Shutdown: %v", err)
        }
        close(idleConnsClosed)
    }()

    if err := srv.ListenAndServe(); err != http.ErrServerClosed {
        // Error starting or closing listener:
        log.Printf("HTTP server ListenAndServe: %v", err)
    }

    <-idleConnsClosed
}

Also you should add Liveness and Readiness probes to your pods:

livenessProbe:
  httpGet:
    path: /health
    port: 80
readinessProbe:
  httpGet:
    path: /health
    port: 80