Common Lisp web app started in Systemd: "compilation unit aborted, caught 1 fatal ERROR condition"

I can start my web app with a make run just fine. When I start it with Systemd I can see the initialization steps succeed, I see the lisp * prompt but instantly after I get this error:

* ;
; compilation unit aborted
; caught 1 fatal ERROR condition

What's going on, what am I missing?

My .service file:

[Unit]
Description=my app

[Service]
WorkingDirectory=/home/vince/projets/myapp/
ExecStart=/usr/bin/make run
Type=simple
Restart=on-failure

My run script does the following:

;; (eval-when (:compile-toplevel :load-toplevel :execute))

(load "myapp.asd")

(unless (ql:quickload "myapp")
  (uiop:quit 1))

(handler-case
    (progn
      (uiop:format! t "-------- start app…") ;; so far so good
      (myapp:startapp :port (ignore-errors (parse-integer (uiop:getenv "PORT")))))

  (error (c)
    (format *error-output* "~&An error occured: ~a~&" c)
    ;; edit:
    (trivial-backtrace:print-backtrace c)))

This starts the Hunchentoot web server.

and I start SBCL with the following switches (not that in makes a difference, with or without the switches it works fine):

LISP ?= /usr/bin/sbcl  --core /home/vince/projets/ciel/ciel --disable-debugger --userinit /home/vince/.sbclrc

status:

$ sudo systemctl status myapp.service     *[master] 
● cmdcollectivites.service - myapp. 
   Loaded: loaded (/etc/systemd/system/myapp.service; static; vendor preset: enabled)
   Active: inactive (dead)

Jun 09 11:22:48 pommier make[31598]: Loading config file...
Jun 09 11:22:48 pommier make[31598]: Starting the web server on port 9999
Jun 09 11:22:48 pommier make[31598]: ✔ Ready. You can access the application!
Jun 09 11:22:48 pommier make[31598]: * ;
Jun 09 11:22:48 pommier make[31598]: ; compilation unit aborted
Jun 09 11:22:48 pommier make[31598]: ;   caught 1 fatal ERROR condition
lines 1-14/14 (END)

There's something that Systemd expects but that CL is not giving. What about the Lisp prompt?

SBCL 1.4.5-debian (some issues with the 2.0…)

Using handler-bind doesn't change the output.

(handler-bind ((error (lambda (c)
                        (format *error-output* "~&An error occured: ~a~&" c)
                        (format *error-output* "~&Backtrace: ~&")
                        (trivial-backtrace:print-backtrace c))))

  (progn
      (uiop:format! t "-------- start app… --------------")
      (cmdcollectivites:startapp :port (ignore-errors (parse-integer (uiop:getenv "PORT"))))))

Solution

I got tricked by the * prompt. I knew what's going on since I wrote about it on StackOverflow: Deploying Common Lisp Web Applications and on the Cookbook: https://lispcookbook.github.io/cl-cookbook/scripting.html#for-web-apps

But in this case I saw this log:

* ;
; compilation unit aborted
; caught 1 fatal ERROR condition

I thought the REPL was waiting for us, and that there was actually an error. Wrong.

The issue was that the Hunchentoot server was started in the background (as always), and nothing was telling the Lisp image to stay with us. It exited right away.

The solution is to put the server thread in the foreground, here using bordeaux-threads:

(bt:join-thread (find-if (lambda (th)
                               (search "hunchentoot" (bt:thread-name th)))
                             (bt:all-threads)))

But still… it isn't clear to me why we get "caught 1 fatal error" with Systemd, why the process stops when it works as expected when run from the terminal, where we have the REPL waiting on the foreground.

Things to keep in mind when starting your app:

you probably rely on Quicklisp. Then on your .sbclrc. Then on QL's snippet which uses (user-homedir-pathname). But you use Systemd from root, and you probably didn't install QL in your root home directory. So, adapt this line, or use this sbcl switch:

sbcl --userinit /path/to/your/.sbclrc
the --disable-debugger switch is good for production (but handy to keep for debugging…)
don't forget you can build a binary, so you won't face these two issues.