I am new to node and following the excellent AWS Developer tutorial by Ryan Lewis. In that course we learn to deploy a Node.js application to AWS EC2 using the Bitnami Node.js image from the AWS Marketplace. To practice a bit I wanted to turn the app into a service using systemd, so that it comes back after a restart. However, after lots of debugging I figured out that the service seems to be restarting all the time and the app never comes online. This might be caused by the way the app is started. It is run using the forever CLI tool. When I run npm start manually, I see the following output:
npm start
> [email protected] prestart /home/bitnami/hamstercourse
> npm run build
> [email protected] build /home/bitnami/hamstercourse
> webpack
(node:19917) DeprecationWarning: Chunk.modules is deprecated. Use Chunk.getNumberOfModules/mapModules/forEachModule/containsModule instead.
Hash: aa4bec1a367d114f2c7f
Version: webpack 3.3.0
Time: 12349ms
Asset Size Chunks Chunk Names
application.min.js 363 kB 0 [emitted] [big] application
stylesheet.css 13.4 kB 0 [emitted] application
[10] ./node_modules/react-redux/es/index.js + 14 modules 37.6 kB {0} [built]
[18] ./node_modules/react-router-dom/es/index.js + 13 modules 11.9 kB {0} [built]
[59] ./node_modules/redux/es/index.js + 6 modules 21.3 kB {0} [built]
[62] ./node_modules/react-router-redux/es/index.js + 4 modules 5.87 kB {0} [built]
[105] ./app/index.jsx 1.86 kB {0} [built]
[209] ./app/router.jsx 2.85 kB {0} [built]
[211] ./app/routes/index.jsx 1.65 kB {0} [built]
[287] ./app/reducers/index.js 316 bytes {0} [built]
[288] ./app/reducers/hamsters.js 717 bytes {0} [built]
[289] ./app/reducers/races.js 649 bytes {0} [built]
[290] ./app/reducers/user.js 2.32 kB {0} [built]
[291] ./app/reducers/leaderboards.js 355 bytes {0} [built]
[292] ./app/reducers/status.js 285 bytes {0} [built]
[293] ./app/index.less 41 bytes {0} [built]
[326] ./node_modules/css-loader?modules&importLoaders=1&localIdentName=[name]__[local]___[hash:base64:5]!./node_modules/less-loader/dist!./app/index.less 247 bytes [built]
+ 312 hidden modules
Child extract-text-webpack-plugin:
[0] ./node_modules/css-loader?modules&importLoaders=1&localIdentName=[name]__[local]___[hash:base64:5]!./node_modules/less-loader/dist!./app/index.less 247 bytes {0} [built]
+ 1 hidden module
Child extract-text-webpack-plugin:
2 modules
Child extract-text-webpack-plugin:
2 modules
Child extract-text-webpack-plugin:
2 modules
Child extract-text-webpack-plugin:
2 modules
Child extract-text-webpack-plugin:
2 modules
Child extract-text-webpack-plugin:
2 modules
Child extract-text-webpack-plugin:
2 modules
Child extract-text-webpack-plugin:
[0] ./node_modules/css-loader?modules&importLoaders=1&localIdentName=[name]__[local]___[hash:base64:5]!./node_modules/less-loader/dist!./app/scenes/Config/index.less 485 bytes {0} [built]
+ 1 hidden module
Child extract-text-webpack-plugin:
[0] ./node_modules/css-loader?modules&importLoaders=1&localIdentName=[name]__[local]___[hash:base64:5]!./node_modules/less-loader/dist!./app/scenes/User/index.less 275 bytes {0} [built]
+ 1 hidden module
Child extract-text-webpack-plugin:
[0] ./node_modules/css-loader?modules&importLoaders=1&localIdentName=[name]__[local]___[hash:base64:5]!./node_modules/less-loader/dist!./app/scenes/Leaderboards/index.less 485 bytes {0} [built]
+ 1 hidden module
Child extract-text-webpack-plugin:
[0] ./node_modules/css-loader?modules&importLoaders=1&localIdentName=[name]__[local]___[hash:base64:5]!./node_modules/less-loader/dist!./app/scenes/Races/index.less 485 bytes {0} [built]
+ 1 hidden module
Child extract-text-webpack-plugin:
[0] ./node_modules/css-loader?modules&importLoaders=1&localIdentName=[name]__[local]___[hash:base64:5]!./node_modules/less-loader/dist!./app/scenes/Race/index.less 485 bytes {0} [built]
+ 1 hidden module
Child extract-text-webpack-plugin:
[0] ./node_modules/css-loader?modules&importLoaders=1&localIdentName=[name]__[local]___[hash:base64:5]!./node_modules/less-loader/dist!./app/scenes/Main/index.less 501 bytes {0} [built]
+ 1 hidden module
Child extract-text-webpack-plugin:
[0] ./node_modules/css-loader?modules&importLoaders=1&localIdentName=[name]__[local]___[hash:base64:5]!./node_modules/less-loader/dist!./app/scenes/Login/index.less 488 bytes {0} [built]
+ 1 hidden module
Child extract-text-webpack-plugin:
[0] ./node_modules/css-loader?modules&importLoaders=1&localIdentName=[name]__[local]___[hash:base64:5]!./node_modules/less-loader/dist!./app/scenes/Hamster/index.less 485 bytes {0} [built]
+ 1 hidden module
Child extract-text-webpack-plugin:
[0] ./node_modules/css-loader?modules&importLoaders=1&localIdentName=[name]__[local]___[hash:base64:5]!./node_modules/less-loader/dist!./app/scenes/Hamsters/index.less 617 bytes {0} [built]
+ 1 hidden module
Child extract-text-webpack-plugin:
2 modules
Child extract-text-webpack-plugin:
2 modules
Child extract-text-webpack-plugin:
2 modules
Child extract-text-webpack-plugin:
2 modules
Child extract-text-webpack-plugin:
2 modules
Child extract-text-webpack-plugin:
2 modules
Child extract-text-webpack-plugin:
2 modules
Child extract-text-webpack-plugin:
2 modules
Child extract-text-webpack-plugin:
[0] ./node_modules/css-loader?modules&importLoaders=1&localIdentName=[name]__[local]___[hash:base64:5]!./node_modules/less-loader/dist!./app/scenes/Main/Hero/index.less 827 bytes {0} [built]
+ 1 hidden module
Child extract-text-webpack-plugin:
2 modules
Child extract-text-webpack-plugin:
2 modules
Child extract-text-webpack-plugin:
2 modules
Child extract-text-webpack-plugin:
2 modules
> [email protected] start /home/bitnami/hamstercourse
> forever stopall && ./node_modules/.bin/forever start index.js
info: No forever processes running
warn: --minUptime not set. Defaulting to: 1000ms
warn: --spinSleepTime not set. Your script will exit if it does not stay up for at least 1000ms
info: Forever processing file: index.js
The process then goes to the background and can be managed using forever list, forever stop, etc. As explained here, systemd kills background processes (and with good reasoning behind it).
To confirm my suspicion I tried running the service like this:
[Unit]
Description=Node.js Hamster Http Server
[Service]
PIDFile=~/hamster-99.pid
User=bitnami
Group=bitnami
Restart=always
RestartSec=10
StandardOutput=syslog
StandardError=syslog
KillSignal=SIGQUIT
WorkingDirectory=/home/bitnami/hamstercourse
ExecStart=/opt/bitnami/nodejs/bin/node /home/bitnami/hamstercourse/index.js
[Install]
WantedBy=multi-user.target
And after enabling the service and reloading the deamon the output of is:
hamster.service - Node.js Hamster Http Server
Loaded: loaded (/etc/systemd/system/hamster.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2019-06-26 10:57:46 UTC; 28min ago
Main PID: 19847 (.node.bin)
Tasks: 11
Memory: 26.8M
CPU: 2.019s
CGroup: /system.slice/hamster.service
+-19847 /opt/bitnami/nodejs/bin/.node.bin /home/bitnami/hamstercourse/index.js
Jun 26 10:57:46 ip-172-31-35-115 systemd[1]: hamster.service: Main process exited, code=dumped, status=3/QUIT
Jun 26 10:57:46 ip-172-31-35-115 systemd[1]: Stopped Node.js Hamster Http Server.
Jun 26 10:57:46 ip-172-31-35-115 systemd[1]: hamster.service: Unit entered failed state.
Jun 26 10:57:46 ip-172-31-35-115 systemd[1]: hamster.service: Failed with result 'core-dump'.
Jun 26 10:57:46 ip-172-31-35-115 systemd[1]: Started Node.js Hamster Http Server.
Jun 26 10:57:48 ip-172-31-35-115 node[19847]: Server started at http://localhost:3000
So the good news is: this actually runs the service. Hurray! However, it skips a lot of essential steps that npm start actually runs (like minifying the app) and it doesn't run the app through forever of course. One may discuss whether it it desirable to use a management tool like forever when running as a service, but it should be possible to run it from systemd without changing the app, right? So how would I do it?
UPDATE:
I just found https://unix.stackexchange.com/questions/308311/systemd-service-runs-without-exiting and tried using Type=forking in the unit file. This actually seems to work. But is it THE WAY? Or is there another best practice?
I think Type=forking is correct.
https://www.freedesktop.org/software/systemd/man/systemd.service.html
If set to forking, it is expected that the process configured with ExecStart= will call fork() as part of its start-up. The parent process is expected to exit when start-up is complete and all communication channels are set up. The child continues to run as the main service process, and the service manager will consider the unit started when the parent process exits. This is the behavior of traditional UNIX services. If this setting is used, it is recommended to also use the PIDFile= option, so that systemd can reliably identify the main process of the service. systemd will proceed with starting follow-up units as soon as the parent process exits.
That is how forever works. Also note that the apache2 service also uses Type=forking.