I have an upstart script for the bitcoind, it based on the script in this topic: https://bitcointalk.org/index.php?topic=25518.0
I strongly need respawn future working: if something happened bitcoind should restart automatically. I tryed to emulate such situation, but upstart didn't restart the process.
Question: how can I make upstart (or something else) watch bitcoind and restart it if something bad happened?
Actual script:
description "bitcoind"
start on filesystem
stop on runlevel [!2345]
oom never
expect daemon
respawn
respawn limit 10 60 # 10 times in 60 seconds
script
user=root
home=/root/.bitcoin/
cmd=/usr/bin/bitcoind
pidfile=$home/bitcoind.pid
# Don't change anything below here unless you know what you're doing
[[ -e $pidfile && ! -d "/proc/$(cat $pidfile)" ]] && rm $pidfile
[[ -e $pidfile && "$(cat /proc/$(cat $pidfile)/cmdline)" != $cmd* ]] && rm $pidfile
exec start-stop-daemon --start -c $user --chdir $home --pidfile $pidfile --startas $cmd -b -m
end script
oom never
is your first problem. You need this:
oom score never
Additionally, do not use oom score never except for critical system services. Try -500 or -700 instead. That should be a higher priority than most processes, but not the ones that are essential for any running system. So you should use:
oom score -500
The second issue is that you are using start-stop-daemon. You should just ditch that, as Upstart can handle everything. So the resulting script would look like this:
description "bitcoind"
start on filesystem
stop on runlevel [!2345]
oom score -500
chdir /root/.bitcoin
respawn
respawn limit 10 60 # 10 times in 60 seconds
exec /usr/bin/bitcoind
The last issue could be that you have not defined normal exit properly. You need to specify which return codes and signals constitute a normal exit, so that Upstart knows to respawn if the signals and return codes do not match. See the Upstart cookbook on how to do this: http://upstart.ubuntu.com/cookbook/#normal-exit.