Let's say we have the following situation:
kill <pid> sends SIGTERM
kill -<SIGNAL> <pid> sends <SIGNAL>
Sometimes, during development, I need to kill my application and restart it, at the moment - using the first type of command. But, if I have a production console opened, I have a chance to kill our production (let's say, I've forgotten about it - THIS HAPPENED RIGHT NOW).
The solution that came into my mind is based on ignoring SIGTERM in production mode, but killing the app gracefully in development mode. This way, if, for some reason, I want to kill our prod, I'll need to specify a SIGNAL to do it, and it'll be impossible to be done accidentally.
The app is built on Twisted.
Twisted has a number useful of methods to use signals with it - for example:
reactor.addSystemEventTrigger('before', 'shutdown', shutdown_callback)
But is it possible to make it ignore a certain signal? I need only one, and I don't want to go this [reactor.run(installSignalHandlers=False)] way (some people say that it doesn't even work) - it'll require me to rewrite the whole signal handling by myself, and that's not what I'm looking for.
Thanks!
This is what I've done in the end (using twistd module with startReactor() override):
def signal_handler(signum, frame):
if signum == signal.SIGTERM:
if is_prod():
log.critical("Received SIGTERM on PRODUCTION call system, ignoring!")
else:
log.critical("Received SIGTERM on DEV call system, shutting down!")
reactor.stop()
elif any([
signum == signal.SIGQUIT,
signum == signal.SIGINT,
signum == signal.SIGILL,
signum == signal.SIGFPE,
signum == signal.SIGABRT,
signum == signal.SIGBUS,
signum == signal.SIGPIPE,
signum == signal.SIGSYS,
signum == signal.SIGSEGV,
signum == signal.SIGHUP
]):
log.critical(f"*Received {signal.Signals(signum).name}, shutting down.*")
reactor.stop()
def register_signals():
signal.signal(signal.SIGTERM, signal_handler)
signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGILL, signal_handler)
signal.signal(signal.SIGFPE, signal_handler)
signal.signal(signal.SIGABRT, signal_handler)
signal.signal(signal.SIGBUS, signal_handler)
signal.signal(signal.SIGPIPE, signal_handler)
signal.signal(signal.SIGSYS, signal_handler)
signal.signal(signal.SIGSEGV, signal_handler)
signal.signal(signal.SIGHUP, signal_handler)
# ...
class ApplicationRunner(twistd._SomeApplicationRunner):
def startReactor(self, reactor, oldstdout, oldstderr):
self._exitSignal = None
from twisted.internet import reactor
try:
reactor.run(installSignalHandlers=False)
except BaseException:
close = False
if self.config["nodaemon"]:
file = oldstdout
else:
file = open("TWISTD-CRASH.log", "a")
close = True
try:
traceback.print_exc(file=file)
file.flush()
finally:
if close:
file.close()
def createOrGetApplication(self):
return application
def run(self):
self.preApplication()
self.application = self.createOrGetApplication()
self.postApplication()
register_signals()
twistd._SomeApplicationRunner = ApplicationRunner
twistd.run()
Basically, this code gets access to the execution of the inner reactor, adds the required parameter, and takes all signal handling on itself. Not the best solution, but that's all we have now.
Known bug: OS kills processes with SIGTERM during the restart, so if the OS triggers the process shutdown, it will send SIGTERM there, and then OS will hang. The solution is to check the following before denying the SIGTERM request:
shutdown, reboot, poweroff, and other stuff like that (Poweruser wants to shut down the server, so we should proceed with the process shut down)