Search code examples
bashdockermonit

Monit doesn't start again with a bash script after docker stop

I have a docker container which uses monit to start a few services like mongodb, nginx. I have a bash script which is used by docker to start monit. Here are the contents of the file:

#!/bin/bash

# Monit will start all apps
/usr/bin/monit -c /etc/monitrc &

# Stay up for container to stay alive
while [ 1 ] ; do
  if !(pgrep monit)
  then /usr/bin/monit -c /etc/monitrc &
  fi
  sleep 5m
done

The problem is when I run the docker create command to create the container, the bash script runs properly and monit brings up all the services, but if I stop the container and start it again, sometimes monit doesn't come up or if it does come up, it doesn't start the services. Could anybody verify whether my bash script is fine. I used the following reference https://blog.deimos.fr/2016/01/13/docker-why-you-should-use-monit-instead-of-supervisord/

Solution

  • To run multiple processes in a container, use an init system that supports running in Docker as PID 1 like s6 via s6-overlay or supervisord. Adding custom scripts in between just adds an extra link in the service chain and the possibility of introducing issues into the system.

    Signals and Docker

    A docker stop sends a SIGTERM signal to the container process, and then after a default of 10 seconds sends a SIGKILL which will terminate the container and anything running in it.

    A process running under docker is PID 1 so needs to handle any signals you want to take action on, otherwise they will be ignored. In the case of a service manager this need to pass the SIGTERM onto all managed services. Note that the example scripts on the blog trap an EXIT of the script, which in turn runs a script to stop all the monit services cleanly. The EXIT includes SIGTERM.

    In your example script, all processes running in the container will be terminated with a SIGKILL after the docker stop times out.

    Services and SIGKILL

    I haven't used monit very much, but as a service manager I would expect it to be able to gracefully handle being killed with a SIGKILL (or a kill -9) and come back up the next time (possibly not though).

    nginx would normally survive a SIGKILL as it doesn't actively store much state on disk by default.

    mongodb does need to be shutdown cleanly. A SIGKILL will leave a database lock file in place that requires manual cleanup.