Sucker Punch Job being killed by Passenger? or Deadlock?

Wondering if anyone has seen this problem. I am running Rails 3.2 on Passenger 3 with sucker_punch gem version 1.1

I have a long running sucker_punch job ( takes around 10 hours) it's an overnight batch. I am running on Phusion Passenger with (I think 3 worker threads)

status from passenger-status
----------- General information -----------
max = 3
count = 0
active = 0
inactive = 0
Waiting on global queue: 0

My sucker_punch job is executed async, as part of the job it executes other async smaller sucker_punch jobs ( each take around 30 seconds)

I cannot exactly determine what is going on, but 'sometimes' my long running job just dies or seems to halt. I did added some debug code around the entire sucker_punch job

begin
rescue Exception => e
  logger.error(e)
  raise e
end

However didn't see an exception, So assuming my long running sucker_punch is being halted rather than killed? Or potential some sort of deadlock?

The interesting part of this. Sometimes my long running job works fine, and sometimes it doesn't.

Solution

This is correct. Currently, Passenger assumes that your processes only handle web requests, not background tasks. Because of this, Passenger enforces certain limits in order to keep buggy apps in check. One of those limits is that if a process is instructed to shutdown, it must do so within 1 minute; if it doesn't, Passenger will force it to shutdown. Unfortunately, this is inherently incompatible with the notion of running background tasks inside app processes.

There's currently an issue open for this: https://github.com/phusion/passenger/issues/1211

Maybe we will work on this in the future, but for now it's not considered a high-priority item. I recommend that you use an out-of-process background worker system, like Sidekiq.