I am using Paperclip and DelayedJob, to process uploaded styles in the background. When I send an image into the queue to be processed, DelayedJob finishes the job, but it gets sent into an infinite loop creating more of the exact same job.
Versions:
delayed_job_active_record version 4.0.2.
paperclip version 4.2.0.
Here is some code plus the worker log to show what is going on...
has_attached_file :data,
url: '/updates/:id/:style/:basename.:extension',
styles: { large: '750x750>', medium: '300x300>', thumb: '50x50#' },
convert_options: { thumb: '-quality 75 -strip' }
validates_attachment_size :data, less_than: 20.megabytes
validates_attachment_content_type :data, content_type: /\Aimage\/.*\Z/
before_data_post_process do |update|
if !update.processing && update.data_changed?
update.processing = true
false # stop processing
end
end
after_save do |update|
if update.processing
update.delay.regenerate_styles!
end
end
def regenerate_styles!
data.reprocess!
update_attribute(:processing, false)
save
end
def data_changed?
data_file_size_changed? ||
data_file_name_changed? ||
data_content_type_changed? ||
data_updated_at_changed?
end
Worker response
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Starting job worker
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=52) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=52) COMPLETED after 2.5247
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=53) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=53) COMPLETED after 0.1654
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=54) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=54) COMPLETED after 0.0895
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=55) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=55) COMPLETED after 0.0993
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=56) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=56) COMPLETED after 0.0990
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=57) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=57) COMPLETED after 0.0947
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=58) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=58) COMPLETED after 0.0949
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=59) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=59) COMPLETED after 0.0855
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=60) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=60) COMPLETED after 0.1004
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=61) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=61) COMPLETED after 0.0941
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=62) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=62) COMPLETED after 0.0860
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=63) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=63) COMPLETED after 0.0980
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=64) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=64) COMPLETED after 0.0928
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=65) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=65) COMPLETED after 0.1053
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=66) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=66) COMPLETED after 0.7533
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=67) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=67) COMPLETED after 0.0950
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=68) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=68) COMPLETED after 0.0952
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=69) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=69) COMPLETED after 0.0867
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=70) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=70) COMPLETED after 0.0944
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=71) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=71) COMPLETED after 0.0913
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=72) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=72) COMPLETED after 0.0918
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=73) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=73) COMPLETED after 0.0982
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=74) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=74) COMPLETED after 0.0976
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=75) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=75) COMPLETED after 0.0925
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=76) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=76) COMPLETED after 0.0914
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=77) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=77) COMPLETED after 0.0970
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=78) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=78) COMPLETED after 0.0922
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=79) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=79) COMPLETED after 1.4655
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=80) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=80) COMPLETED after 0.0892
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=81) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=81) COMPLETED after 0.0839
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=82) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=82) COMPLETED after 0.0848
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=83) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=83) COMPLETED after 0.0965
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=84) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=84) COMPLETED after 0.0963
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=85) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=85) COMPLETED after 0.0917
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=86) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=86) COMPLETED after 0.0919
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=87) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=87) COMPLETED after 0.0909
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=88) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=88) COMPLETED after 0.1006
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=89) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=89) COMPLETED after 0.0936
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=90) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=90) COMPLETED after 0.2311
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=91) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=91) COMPLETED after 0.1069
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=92) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=92) COMPLETED after 0.6727
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=93) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=93) COMPLETED after 0.0904
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=94) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=94) COMPLETED after 0.0993
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=95) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=95) COMPLETED after 0.0903
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=96) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=96) COMPLETED after 0.0976
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=97) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=97) COMPLETED after 0.0916
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=98) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=98) COMPLETED after 0.1023
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=99) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=99) COMPLETED after 0.0991
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=100) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=100) COMPLETED after 0.0997
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=101) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=101) COMPLETED after 0.0953
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=102) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=102) COMPLETED after 0.0933
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=103) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=103) COMPLETED after 0.0918
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=104) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=104) COMPLETED after 0.6735
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=105) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=105) COMPLETED after 0.0884
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=106) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=106) COMPLETED after 0.0916
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=107) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=107) COMPLETED after 0.0859
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=108) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=108) COMPLETED after 0.0886
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=109) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=109) COMPLETED after 0.0871
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=110) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=110) COMPLETED after 0.0853
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=111) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=111) COMPLETED after 0.0991
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=112) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=112) COMPLETED after 0.0889
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=113) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=113) COMPLETED after 0.0969
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=114) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=114) COMPLETED after 0.0956
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=115) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=115) COMPLETED after 0.0993
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=116) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=116) COMPLETED after 0.0987
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=117) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=117) COMPLETED after 0.6886
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=118) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=118) COMPLETED after 0.0927
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=119) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=119) COMPLETED after 0.0962
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=120) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=120) COMPLETED after 0.0991
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=121) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=121) COMPLETED after 0.0924
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=122) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=122) COMPLETED after 0.0938
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=123) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=123) COMPLETED after 0.0965
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=124) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=124) COMPLETED after 0.0949
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=125) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=125) COMPLETED after 0.0933
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=126) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=126) COMPLETED after 0.0950
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=127) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=127) COMPLETED after 0.0877
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=128) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=128) COMPLETED after 0.0954
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=129) RUNNING
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=129) COMPLETED after 0.8408
[Worker(host:Justins-MacBook-Pro.local pid:98244)] Job Update#regenerate_styles! (id=130) RUNNING
It probably takes 10 seconds for DJ to run through all those jobs as well. Has anyone run into this issue or can anyone spot an error with my configuration? Let me know if any more code is needed. Thanks in advance.
You call data.reprocess! with a bang, this probably saves the model (before you set processing to false) which then with the after_save generates another job.
Your data changed method also checks updated_at (like you commented below) calling any kind of save (also the "processing, false" save) will update the updated_at. (causing processing again to be turned on)
Check the second answer here: How to update a single attribute without touch updated_at attribute?
and use update_column to skip the updating of the updated_at column.
thus:
def regenerate_styles!
update_column(:processing, false)
data.reprocess!
end