I run 2 nodJS servers on a Linux-Machine. In order to keep them running even when I close the console I start them with screen. This all works perfectly. As long as my Linux-Server-provider doesn't do any maintenance work. Then I have to restart the servers manually.
I am currently trying to harden my NodeJS-Servers against being down after a server maintenance of my Linux-Server-Provider. The node servers are supposed to restart automatically when the Linux-Server is back up again.
Following some of the many articles on the Internet I was able to create a systemd Service which is located in /etc/systemd/system/.
I can see the service being correctly enabled and loaded in systemd when I run systemctl list-units --all | grep my1984.service
I also can see its status when I fire sudo systemctl status my1984.service.
This all works pretty fine. But now comes the interesting part of it.
This service calls a shell script. And doing so it only executes the contained echo commands. All the other commands are being ignored. And it doesn't matter whether it is run after a server reboot or when I start this service manually via sudo systemctl start my1984.service. But when I run the shell script itself with bash ./my1984.sh the script gets run properly, meaning that all echo commands as well as all the other commands are getting processed as expected.
I tried to log error messages into an error file but no errors appear. It seems that the service simply ignores all the other commands but echo. And it does not matter whether I call the service with root or with my local user that I granted some extra right in sudoers.d in order to being able to use systemctl.
Here is the content of my service:
[Unit]
Description=My 1984 Service Test
[Service]
ExecStart=/bin/bash /home/my1984/scripts/my1984.sh
[Install]
WantedBy=multi-user.target
This is the content of my shell script
#!/bin/bash
echo $(date -u) "executing my1984.sh." >> /home/my1984/scripts/scripts.log
screen -ls >> scripts.log
screen -S NODE-Server 1 -d -m node /home/my1984/node-server1/bin/www 2>> error.log
echo $(date -u) "starting screen node @ node-server1." >> /home/my1984/scripts/scripts.log
screen -S NODE-Server 2 -d -m node /home/my1984/node-server2/bin/www 2>> error.log
echo $(date -u) "starting screen node @ node-server2." >> /home/my1984/scripts/scripts.log
echo $(date -u) "my1984.sh has been executed." >> /home/my1984/scripts/scripts.log
As I said, my local user my1984 has been given some sudo rights in order to be able to run certain systemctl commands in combination with sudo.
When I call the script locally with the bash-command it works properly and screen -ls shows the node servers in its list. But when the my1984.service calls the script only the echo-commands are being processed and deliver output into the file. screen -ls shows no content. No errors are being written to the error.log in this case.
What can be the cause of this strange behaviour?
Here is some technical information about my Linux Server and so on
Linux
NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.4 LTS"
VERSION_ID="18.04"
Screen
Screen version 4.06.02 (GNU) 23-Oct-17
Node
v17.9.1
It is bad practice to start a double forking process (bash starting screen starting node) if it can be avoided.
Easiest solution would be to just create two services, one for each node server.
I run multiple discord bots and nodejs applications, and I am doing it this exact same way.
I know, this is not the solution for you asked question, but it is the solution for your actual problem :)
Starting and stopping the servers would be done with systemctl, and you can observe the console output of both processes with journalctl
To accomplish this, move your nodejs scripts to a more approriate place. I like to use /srv but /opt would be fine as well.
For this example I will use /opt/servers
Just use something like this as your service file
/etc/systemd/system/noderserver1.service:
[Unit]
Description=1984 Server 1
[Service]
ExecStart=/usr/bin/node /opt/servers/node-server1/bin/www
[Install]
WantedBy=multi-user.target
do the same for server2
you would access the logs with journalctl -u nodeserver1.service (read up on other parameters like -f or -e of journalctl)