When sshing into the webserver I can restart delayed_job all day without a problem. It brings down the existing worker, starts a new one and writes tmp/pids/delayed_job.pid with its process id. (I am also restarting passenger to mimic what I'm about to do with capistrano)
app@StagingServer:/app/current$ touch tmp/restart.txt; RAILS_ENV=staging script/delayed_job restart
delayed_job: trying to stop process with pid 22170...
delayed_job: process with pid 22170 successfully stopped.
delayed_job: process with pid 22284 started.
app@StagingServer:/app/current$ touch tmp/restart.txt; RAILS_ENV=staging script/delayed_job restart
delayed_job: trying to stop process with pid 22284...
delayed_job: process with pid 22284 successfully stopped.
delayed_job: process with pid 22355 started.
app@StagingServer:/app/current$ touch tmp/restart.txt; RAILS_ENV=staging script/delayed_job restart
delayed_job: trying to stop process with pid 22355...
delayed_job: process with pid 22355 successfully stopped.
delayed_job: process with pid 22427 started.
app@StagingServer:/app/current$
However, when I deploy using capistrano
dev@ubuntu:~/app-site$ cap passenger:restart
triggering start callbacks for `passenger:restart'
* executing `multistage:ensure'
*** Defaulting to `staging'
* executing `staging'
* executing `passenger:restart'
* executing "touch /app/current/tmp/restart.txt"
servers: ["staging.app.com"]
[staging.app.com] executing command
command finished in 242ms
* executing "cd /app/current;RAILS_ENV=staging script/delayed_job restart"
servers: ["staging.app.com"]
[staging.app.com] executing command
** [out :: staging.app.com] delayed_job: trying to stop process with pid 21646...
** [out :: staging.app.com] delayed_job: process with pid 21646 successfully stopped.
command finished in 11889ms
Seems fine? While the last line from delayed_job isn't printed (I think due to it not ending in a newline) this does successfully create a new process. However, it doesn't create the .pid file, so when I try and restart again:
dev@ubuntu:~/app-site$ cap passenger:restart
triggering start callbacks for `passenger:restart'
* executing `multistage:ensure'
*** Defaulting to `staging'
* executing `staging'
* executing `passenger:restart'
* executing "touch /app/current/tmp/restart.txt"
servers: ["staging.app.com"]
[staging.app.com] executing command
command finished in 398ms
* executing "cd /app/current;RAILS_ENV=staging script/delayed_job restart"
servers: ["staging.app.com"]
[staging.app.com] executing command
** [out :: staging.app.com] Warning: no instances running. Starting...
** [out :: staging.app.com] delayed_job: process with pid 21950 started.
command finished in 6758ms
It doesn't stop the existing process. Strangely, this time it will create a new process and it's .pid file.
This leaves me with 2 delayed_jobs processes running and only one in the .pid file. Every 2 cap deploys I do will add another delayed_job process. The previous processes are using the old version of the app, essentially breaking it.
config/deploy.rb:
namespace :passenger do
desc "Restart Application"
task :restart do
run "touch #{current_path}/tmp/restart.txt"
run "cd #{current_path};RAILS_ENV=#{deploy_env} script/delayed_job restart"
end
end
after :deploy, "passenger:restart"
And locally
Update:
Reading around, it seems it could be to do with a race condition inside daemons. However I'm a bit confused why it's showing only (and consistently) when using capistrano. I will try changing the command to a stop, sleep then start.
Solved by using stop and start instead of restart. Works due to a race condition probably caused by the daemons gem.
I would still like to know if anyone else has a better solution.