Search code examples
systemd

systemd unit file never finishes when using a shell command with pipes


I'm trying to execute a shell script with systemd.

If I run my script from the bash everything works fine. But if I run the same script via systemd it never finishes. The command where it seems to hang is:

random="$(LC_ALL=C tr -cd '[:alnum:]' < /dev/urandom | fold -w128 | head -n1)"

If I'm replacing this line with random="1234" it also runs with systemd. I guess the 'hanging' command is the tr - its process never finishes.

And this is the systemd unit file I'm using:

[Unit]
Description=my script

[Service]
Type=forking
Restart=on-failure
ExecStart=/mypath/script.sh start
ExecStop=/bin/kill $MAINPID
ExecStopPost=/bin/rm -f /mypath/RUNNING_PID

[Install]
WantedBy=multi-user.target

Solution

  • Edit: Made the explanation clearer and added new information.

    Short answer: systemd is filtering SIGPIPEs, so set IgnoreSIGPIPE=false under [Service] in the .service file. From the systemd.exec manual:

       IgnoreSIGPIPE=
           Takes a boolean argument. If true, causes
           SIGPIPE to be ignored in the executed process.
           Defaults to true because SIGPIPE generally is
           useful only in shell pipelines.
    

    Long explanation:

    random="$(LC_ALL=C tr -cd '[:alnum:]' < /dev/urandom | fold -w128 | head -n1)"

    When the head command exits after the first newline received from fold, its open file descriptors are closed. Thus when the fold command later tries to write to the pipe, it will receive a SIGPIPE signal (as described in pipe(7)). The default action for this signal is a termination of the process. That mechanism repeats towards the start of the pipeline, so the fold command would be terminated next, and eventually the tr command.

    However, when the pipeline is run under systemd, systemd sets the default action for SIGPIPE to SIG_IGN, which makes the processes in the pipeline ignore the signal.

    When fold (and the other components of the pipeline) inherits that signal handler, it won't be notified (killed) via SIGPIPE when writing to the now closed output pipe (Instead each write to the pipe will return an EPIPE error).

    As fold command does not check the return value of those write calls (at least not in coreutils-8.26), this makes fold continue reading from the pipe (stdin) and to write to the closed pipe (stdout), even though an error is reported each time. So fold keeps its input pipe from tr open, and tr will happily write to the output pipe feeding fold forever.