Search code examples
capistranosidekiqmonit

Setting up deployments with Capistrano, Sidekiq and Monit


My application uses Sidekiq to handle long (several minutes) running background tasks. Deployments are done with Capistrano 2 and all processes are monitored with Monit.

I have used capistrano-sidekiq to manage the sidekiq process during deployments but it has not worked perfectly. Some times during the deployment a new sidekiq process is started but the old one is not killed. I believe this happens because capistrano-sidekiq is not operating through Monit during the deployment.

Second problem is that because my background tasks can take several minutes to complete my deployment should allow two sidekiq processes to co-exisit. The old sidekiq process should be allowed to complete the tasks it is processing and a new sidekiq process should start taking new tasks into processing.

I have been thinking about something like this into my deploy script

When deployment starts:

  • I tell Monit to unmonitor the sidekiq process
  • I stop the current sidekiq process and give it 10 minutes to finish its tasks

After the code has been updated:

  • I start a new sidekiq process and tell Monit to start monitoring it.

I may need to move the sidekiq process pid file into the release directory if the pid file is not removed until the stopped sidekiq process has eventually been killed.

How does this sound? Any caveats spotted?

EDIT:

Found a good thread about this same issue.

http://librelist.com/browser//sidekiq/2014/6/5/rollback-signal-after-usr1/#f6898deccb46801950f40ad22e75471d


Solution

  • Seems reasonable to me. The only possible issue is losing track of the old Sidekiq's PID but you should be able to use ps and grep for "stopping" to find old Sidekiqs.