Using Supervisord to manage mongos process

Question

background

I am trying to automate the restarting in case of crash or reboot for mongos process used in mongodb sharded setup.

Case 1 : using direct command, with mongod user

supervisord config

[program:mongos_router]
command=/usr/bin/mongos -f /etc/mongos.conf --pidfilepath=/var/run/mongodb/mongos.pid
user=mongod
autostart=true
autorestart=true
startretries=10

Result

supervisord log

INFO spawned: 'mongos_router' with pid 19535
INFO exited: mongos_router (exit status 0; not expected)
INFO gave up: mongos_router entered FATAL state, too many start retries too quickly    

mongodb log

  2018-05-01T21:08:23.745+0000 I SHARDING [Balancer] balancer id: ip-address:27017 started
  2018-05-01T21:08:23.745+0000 E NETWORK  [mongosMain] listen(): bind() failed errno:98 Address already in use for socket: 0.0.0.0:27017
  2018-05-01T21:08:23.745+0000 E NETWORK  [mongosMain]   addr already in use
  2018-05-01T21:08:23.745+0000 I -        [mongosMain] Invariant failure inShutdown() src/mongo/db/auth/user_cache_invalidator_job.cpp 114
  2018-05-01T21:08:23.745+0000 I -        [mongosMain] 

  ***aborting after invariant() failure


  2018-05-01T21:08:23.748+0000 F -        [mongosMain] Got signal: 6 (Aborted).

Process is seen running. But if killed does not restart automatically.

Case 2 : Using init script

here the slight change in the scenario is that some ulimit commands, creation of pid files is to be done as root and then the actual process should be started as mongod user.

mongos script

start()
{
  # Make sure the default pidfile directory exists
  if [ ! -d $PID_PATH ]; then
    install -d -m 0755 -o $MONGO_USER -g $MONGO_GROUP $PIDDIR

  fi

  # Make sure the pidfile does not exist
  if [ -f $PID_FILE ]; then
    echo "Error starting mongos. $PID_FILE exists."
    RETVAL=1
    return
  fi

  ulimit -f unlimited
  ulimit -t unlimited
  ulimit -v unlimited
  ulimit -n 64000
  ulimit -m unlimited
  ulimit -u 64000
  ulimit -l unlimited

  echo -n $"Starting mongos: "
  #daemon --user "$MONGO_USER" --pidfile $PID_FILE $MONGO_BIN $OPTIONS --pidfilepath=$PID_FILE
  #su $MONGO_USER -c "$MONGO_BIN -f $CONFIGFILE --pidfilepath=$PID_FILE  >> /home/mav/startup_log"
  su - mongod -c "/usr/bin/mongos -f /etc/mongos.conf --pidfilepath=/var/run/mongodb/mongos.pid"

  RETVAL=$?
  echo -n  "Return value : "$RETVAL
  echo
  [ $RETVAL -eq 0 ] && touch $MONGO_LOCK_FILE
}

daemon comman represents original script, but daemonizing under the supervisord is not logical, so using command to run the process in foreground(?)

supervisord config

[program:mongos_router_script]
command=/etc/init.d/mongos start
user=root
autostart=true
autorestart=true
startretries=10

Result

supervisord log

INFO spawned: 'mongos_router_script' with pid 20367     
INFO exited:  mongos_router_script (exit status 1; not expected)
INFO gave up: mongos_router_script entered FATAL state, too many start retries too quickly

mongodb log

Nothing indicating error, normal logs

Process is seen running. But if killed does not restart automatically.

Problem

How to correctly configure script / no script option for running mongos under supervisord ?

EDIT 1

Modified Command

sudo su -c "/usr/bin/mongos -f /etc/mongos.conf --pidfilepath=/var/run/mongodb/mongos.pid" -s /bin/bash mongod`

This works if ran individually on command line as well as part of the script, but not with supervisord

EDIT 2

Added following option to config file for mongos to force it to run in the foreground

processManagement:
    fork: false # fork and run in background

Now command line and script properly run it in the foreground but supervisord fails to launch it. At the same time there are 3 processes show up when ran from command line or script

root   sudo su -c /usr/bin/mongos -f /etc/mongos.conf --pidfilepath=/var/run/mongodb/mongos.pid -s /bin/bash mongod
root   su -c /usr/bin/mongos -f /etc/mongos.conf --pidfilepath=/var/run/mongodb/mongos.pid -s /bin/bash mongod
mongod /usr/bin/mongos -f /etc/mongos.conf --pidfilepath=/var/run/mongodb/mongos.pid

EDIT 3

With following supervisord config things are working fine. But I want to try and execute the script if possible to set ulimit

[program:mongos_router]
 command=/usr/bin/mongos -f /etc/mongos.conf --pidfilepath=/var/run/mongodb/mongos.pid
 user=mongod
 autostart=true
 autorestart=true
 startretries=10
 numprocs=1

Answer 1

For the mongos to run in the foreground set the following option

#how the process runs
processManagement:
    fork: false  # fork and run in background

with that and above supervisord.conf setting, mongos will be launched and under the supervisord control