We use Asgard to do our deploys to a Rails 4 application using Sidekiq. We are wondering what happens to jobs that are in the queue when the deploy is ongoing and terminates the worker nodes.
What is the shutdown process for Sidekiq? Does it re-enqueue the jobs that won't get finished or will I lose the state of the running jobs?
If a job will not finished in 8 sec after stop signal - it will be pushed in the Redis and restarted later. Sidekiq wiki:
By default, Sidekiq gives workers 8 seconds to shut down. This is carefully chosen because Heroku gives a process 10 seconds to shutdown before killing it. After 8 seconds, any remaining jobs still in progress are pushed back onto Redis so they can be immediately restarted when Sidekiq starts back up. Remember that Sidekiq will run your jobs AT LEAST once so they need to be idempotent. This is one example of how a job can be run twice.