Search code examples
pythonlinuxmultithreadingsystemd

Systemd not receiving SIGTERM on stop when using threads in program


I want to create a python program which runs as a systemd service. I want to be able to stop it gracefully. I've come across a weird behavior: when I use threads, a python program does not receive a SIGTERM signal on systemctl stop example.service, but everything works fine if I do not use threads. The example is following:

Without threads. (service receives SIGTERM signal and stops as expected):

import signal
import time
import threading
import sys

RUN=True

# catch SIGINT and SIGTERM and stop application
def signal_handler(sig, frame):
    global RUN
    print("Got signal: "+str(sig))
    RUN=False
signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGTERM, signal_handler)

# some working thread inside application
def my_thread():
    global RUN
    while RUN:
        print("sleep")
        time.sleep(1.0)

my_thread()
print("Done.")

stopping a program without threads works

With threads. (program does not receive SIGTERM signal and is forcefully killed by SIGKILL after timeout):

import signal
import time
import threading
import sys

RUN=True

# catch SIGINT and SIGTERM and stop application
def signal_handler(sig, frame):
    global RUN
    print("Got signal: "+str(sig))
    RUN=False
signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGTERM, signal_handler)

# some working thread inside application
def my_thread():
    global RUN
    while RUN:
        print("sleep")
        time.sleep(1.0)

# wait for thread to complete and exit
t = threading.Thread(target=my_thread)
t.start()
t.join()
print("Done.")

stopping a program with threads does not work

Systemd service file:

[Unit]
Description=Example service

[Install]
WantedBy=multi-user.target

[Service]
ExecStart=/usr/bin/python /opt/program/main.py
TimeoutSec=60
Restart=on-failure
Type=simple
User=mixo
Group=mixo

Just to be clear: my program requires multiple threads so I want to be able to stop service gracefully even if I use threads in my program. What am I doing wrong?


Solution

  • Thanks to @Shawn who suggested this old post, I have now solved the problem.

    The problem is in how signal handlers are implemented in python. The line t.join() is blocking my main thread and therefore no signals can be received. There are 2 simple solutions:

    1) Use python 3.x

    or 2) Use signal.pause() to wait for signal like this:

    import signal
    import time
    import threading
    import sys
    
    RUN=True
    
    # catch SIGINT and SIGTERM and stop application
    def signal_handler(sig, frame):
        global RUN
        print("Got signal: "+str(sig))
        RUN=False
    signal.signal(signal.SIGINT, signal_handler)
    signal.signal(signal.SIGTERM, signal_handler)
    
    # some working thread inside application
    def my_thread():
        global RUN
        while RUN:
            print("sleep")
            time.sleep(1.0)
    
    # wait for thread to complete and exit
    t = threading.Thread(target=my_thread)
    t.start()
    signal.pause()
    t.join()
    print("Done.")