How to handle GCP preemptive shutdown for jupyter notebooks

Question

In preemptive VM instances in Google Cloud Platform, a forced shut down can be called at any time. They allow to run a shutdown script to avoid file loss. But how do I use the script to cause a specific interrupt in my jupyter notebook?

Answer 1

I have come up with a solution.

from os import getpid, kill
from time import sleep
import signal
import ipykernel
import psutil


def get_active_kernels():
    active_kernels = []
    pids = psutil.pids()
    my_pid = getpid()

    for pid in pids:
        if pid == my_pid:
            continue
        try:
            p = psutil.Process(pid)
            cmd = p.cmdline()
            for arg in cmd:
                if arg.count('ipykernel'):
                    active_kernels.append(pid)
        except psutil.AccessDenied:
            continue
    return active_kernels

if __name__ == '__main__':
    kernels = get_active_kernels()
    for kernel in kernels:
        kill(kernel, signal.SIGINT)

One can use this code as a shut-down script. It invokes a keyboard interrupt to all existing jupyter kernels. So, a simple try-except block that excepts KeyboardInterrupt can be used inside the jupyter notebook.

try:
    ... #regular code
except KeyboardInterrupt:
    ... #code to save the progress