I have developed a quite big django app.
I have deployed it at path /var/www/mybig_app/ on a server machine (raspberry pi) connected to my LAN.
The app uses libraries stored in the virtual environment /var/www/mybig_app/venv/.
I have configured a .wsgi file to instruct gunicorn (version 19.9.0) to expose the app on port 8003 of localhost and nginx to stream content form port 8003 to port 3003.
So, with current directory being /var/www/mybig_app/, I run
python manage.py runserver localhost:8003
or
/var/www/mybig_app/venv/bin/gunicorn mybig_app.wsgi:application --bind localhost:8003
and even if I have to wait about 5 seconds, in both cases I can reach my app from a browser of another device connected to my LAN at address localhost:8003 (I have tryed to login as admin, access all the pages, it works, there is no error).
Now I want to manage my app as a service via systemd.
So I have prepared a .service file at /var/www/mybig_app/infrastructure/systemd/mybig_app.service, symbollically linked with
ln -s /var/www/mybig_app/infrastructure/systemd/mybig_app.service /etc/systemd/system/
ln -s /var/www/mybig_app/infrastructure/systemd/mybig_app.service /etc/systemd/system/multi-user.target.wants/
loaded the configurations for systemd
sudo systemctl daemon-reload
sudo systemctl enable mybig_app.service
(I have done this before for other apps, it works)
In the end I want to start it, but when from directory /var/www/mybig_app I run
sudo systemctl start mybig_app.service
I get the command holding on, after the 15s timeout I have set is reached, and I get
Job for mybig_app.service failed because a timeout was exceeded.
See "systemctl status mybig_app.service" and "journalctl -xe" for details.
So I run
journalctl -xe
and I see
--
-- A stop job for unit mybig_app.service has finished.
--
-- The job identifier is 20135 and the job result is done.
May 08 17:48:02 Raspberry100 systemd[1]: Starting service for starting the app mybig_app on RPi...
-- Subject: A start job for unit mybig_app.service has begun execution
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit mybig_app.service has begun execution.
--
-- The job identifier is 20135.
May 08 17:48:05 Raspberry100 sudo[1806]: root : TTY=pts/1 ; PWD=/var/www/mybig_app ; USER=root ; COMMAND=/bin/systemctl daemon-reload
May 08 17:48:05 Raspberry100 sudo[1806]: pam_unix(sudo:session): session opened for user root by pi(uid=0)
May 08 17:48:05 Raspberry100 systemd[1]: Reloading.
May 08 17:48:05 Raspberry100 sudo[1806]: pam_unix(sudo:session): session closed for user root
May 08 17:48:09 Raspberry100 sudo[1838]: root : TTY=pts/1 ; PWD=/var/www/mybig_app ; USER=root ; COMMAND=/bin/systemctl start mybig_app.
May 08 17:48:09 Raspberry100 sudo[1838]: pam_unix(sudo:session): session opened for user root by pi(uid=0)
May 08 17:48:18 Raspberry100 systemd[1]: mybig_app.service: Start operation timed out. Terminating.
May 08 17:48:18 Raspberry100 systemd[1]: mybig_app.service: Failed with result 'timeout'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit mybig_app.service has entered the 'failed' state with result 'timeout'.
May 08 17:48:18 Raspberry100 systemd[1]: Failed to start service for starting the app mybig_app on RPi.
-- Subject: A start job for unit mybig_app.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit mybig_app.service has finished with a failure.
--
-- The job identifier is 20135 and the job result is failed.
May 08 17:48:18 Raspberry100 sudo[1838]: pam_unix(sudo:session): session closed for user root
What is the problem here?
I think it is somehow linked to the content of my mybig_app.service file, but I have used the same procedure and the same for other apps in the past, just changing the project name, and it always worked.
One other problem might be the libraries installed, which are quite old (project is from year 2018).
Note that since the app required some seconds (no more than 5) to setup as I runned it via python and gunicorn, in my .service file I have set
[Service]
...
# RestartSec=100ms # this is the default
RestartSec=15s
TimeoutStartSec=15s
TimeoutSec=15s
in order to let it more time to setup (default restart timeout is 0.1 seconds).
I have deleted and recreated the virtual environment.
I have rebooted the server machine.
I have tryed to increase timeout to 115 seconds.
I have noticed that, as the command of starting holds on, I can reach and navigate my app from the browser of another device connected to my LAN.
The issue is similar to this one, since I cannot start or restart my application as service, but I can run it successfully via python or gunicorn. Yet in my case I am not using httpd as web server, but nginx, which works fine for other applications configured the same way.
mybig_app.service
[Unit]
# the first line must be
# [Unit]
#------------------------------------------------------------------------
# what to do with this file
# this file must be named
# <target_app>.service
# and deployed to
# /etc/systemd/system
# to be ran on boot.
# Once in the folder make the file executable
# sudo chmod +x <target_app>.service
# This will start the service but will not run it on boot.
# sudo systemctl start <target_app>.service
# To make your service automatically run on boot
# sudo systemctl daemon-reload
# sudo systemctl enable <target_app>.service
# Run this in terminal to disable the program on boot
# sudo systemctl daemon-reload
# sudo systemctl disable <target_app>.service
# Documentation https://www.freedesktop.org/software/systemd/man/systemd.service.html
#------------------------------------------------------------------------
Description=service for starting the app mybig_app on RPi
After=network.target
# This will not start execution of this file until the network connection is made
# It can be replaced with other parameters of your choosing
# e.g.
# After=syslog.target
[Service]
WorkingDirectory=/var/www/mybig_app/
ExecStart=/var/www/mybig_app/venv/bin/gunicorn mybig_app.wsgi:application --bind localhost:8003
StandardOutput=file:/var/log/mybig_app/mybig_app-stdout.log
StandardError=file:/var/log/mybig_app/mybig_app-stderr.log
#Optional: Saves the output and error log of the terminal to a .log file in a directory of your choosing.
Restart=always
# Automatically restart on kill
# RestartSec=100ms # this is the default
RestartSec=15s
TimeoutStartSec=15s
TimeoutSec=15s
KillSignal=SIGQUIT
# Optional: To cleanly end the file on stop use this command. This sends a terminal interrupt command on the executable script
# alternatively:
# Restart=on-failure
Type=notify
NotifyAccess=all
[Install]
WantedBy=multi-user.target
When you use Type=notify systemd waits for a notification from the service via sd_notify, support for this was only added in Gunicorn in version 20.0 (See the following comment on a GitHub issue).
If you do want to use Type=notify you should upgrade your Gunicorn version, otherwise you can simply set Type=exec.