Start a process from a systemd service that won't be killed when the service is stopped

Question

I have a .NET 6 application that runs as a systemd service on Linux. This application executes another program with a command like the following:

var process = Process.Start(new ProcessStartInfo
    {
      FileName = "foo",
      Arguments = "bar",
    });

This process should continue to run even when this application that started it is terminated. Which as far as I understand is what Process.Start should do. But in my case I'm observing now that the spawned process is getting terminated at the same time as the original process when I stop the original process via systemctl stop.

Both get stopped within the same millisecond, I added a handler for the AppDomain.CurrentDomain.ProcessExit event to both that gets triggered, which does indicate to me that they both get a SIGTERM from systemd.

Now, I assume that systemd does something else here to clean up the service as usually my spawned process should live on when the original dies. How can I start a process in a way that won't be killed by systemd when the parent is stopped?

Answer 1

Systemd by design cleans up all left-over processes within a service's cgroup when that service is supposed to be stopped – it will send the configured KillSignal (SIGTERM) and will follow up with SIGKILL a few seconds later if necessary.

There is no Start() option to have your process avoid the cleanup from within your C# code. At systemd .service unit level, services can partially opt out of cleanup using KillMode=, but there are plans to eventually remove this option.

The correct way to start a process that needs to outlast your service is to make it no longer part of the service – that is, use either the systemd D-Bus API or the systemd-run tool to either a) have systemd itself start the process as its own transient service, or b) move the process that you just started into its own transient .scope unit. (The latter can't be done via systemd-run, only via the D-Bus API.)

For example, using Tmds.DBus (which appears to be more actively maintained than dbus-sharp), the code to detach a PID to its own .scope might look somewhat like this:

var bus = Tmds.DBus.Connection.System;
var sd = bus.CreateProxy<IHaveNoIdea>("org.freedesktop.systemd1",
                                      "/org/freedesktop/systemd1");
var properties = /* I don't remember how to create dicts in C# */;
properties.Add("Description", "FooBar worker process");
properties.Add("PIDs", new int[] { pid_of_worker });
properties.Add("CollectMode", "inactive-or-failed");
var aux = /* empty array? */;
var job = sd.StartTransientUnit("foobar-worker.scope", "fail", properties, aux);

(This is superficially based on the code used by GNOME in gnome-desktop to detach launched apps from gnome-shell.service.)

A new .service unit can be started in a similar way, but with ExecStart instead of PIDs (plus various environment properties such as User or WorkingDirectory that you might want systemd to set up for the process).

Alternatively, you could define a normal .service unit that your main program could start – this would work if the process in question only needs to receive at most one option, which could be passed as service instance name.