I have a free heroku account where I have deployed my rails app. I have also enabled the redis heroku addon. My app scrapes sites and in my local env it took 93 mins to complete. So i decided to put it in background task and run it there. I followed some tutorials to get background task running in free heroku account so I used unicorn server. While testing it locally my app works fine but once I deploy it to heroku and browse my project site it does not work.Here is the result from heroku log
2016-09-16T17:40:40.934711+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:40.943447+00:00 app[web.1]: E, [2016-09-16T17:40:40.943337 #3] ERROR -- : reaped #<Process::Status: pid 13987 exit 1> worker=2
2016-09-16T17:40:40.951820+00:00 app[web.1]: E, [2016-09-16T17:40:40.946468 #13992] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:40.951822+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:40.951824+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:40.951824+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:40.951825+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:40.951826+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:40.951826+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:40.951827+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:40.951828+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:40.951829+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:40.961433+00:00 app[web.1]: E, [2016-09-16T17:40:40.961347 #3] ERROR -- : reaped #<Process::Status: pid 13992 exit 1> worker=1
2016-09-16T17:40:40.966761+00:00 app[web.1]: E, [2016-09-16T17:40:40.953957 #13996] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:40.966764+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:40.966765+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:40.966766+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:40.966767+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:40.966767+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:40.966768+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:40.966769+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:40.966770+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:40.966771+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:40.968861+00:00 app[web.1]: E, [2016-09-16T17:40:40.944646 #13990] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:40.968863+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:40.968864+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:40.968865+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:40.968866+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:40.968867+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:40.968872+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:40.968871+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:40.968873+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:40.977007+00:00 app[web.1]: E, [2016-09-16T17:40:40.976925 #3] ERROR -- : reaped #<Process::Status: pid 13990 exit 1> worker=0
2016-09-16T17:40:40.968873+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:40.977910+00:00 app[web.1]: E, [2016-09-16T17:40:40.965528 #13999] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:40.977913+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:40.977914+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:40.977914+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:40.977915+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:40.977916+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:40.977917+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:40.977918+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:40.977918+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:40.977919+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:40.981913+00:00 app[web.1]: E, [2016-09-16T17:40:40.981796 #3] ERROR -- : reaped #<Process::Status: pid 13996 exit 1> worker=2
2016-09-16T17:40:40.982020+00:00 app[web.1]: E, [2016-09-16T17:40:40.981968 #3] ERROR -- : reaped #<Process::Status: pid 13999 exit 1> worker=1
2016-09-16T17:40:40.998699+00:00 app[web.1]: E, [2016-09-16T17:40:40.985434 #14004] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:40.998703+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:40.998704+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:40.998705+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:40.998706+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:40.998707+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:40.998708+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:40.998708+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:40.998709+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:40.998710+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:41.013015+00:00 app[web.1]: E, [2016-09-16T17:40:41.012891 #3] ERROR -- : reaped #<Process::Status: pid 14004 exit 1> worker=0
2016-09-16T17:40:41.013269+00:00 app[web.1]: E, [2016-09-16T17:40:40.992746 #14009] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:41.013271+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:41.013272+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:41.013272+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:41.013273+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:41.013274+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:41.013275+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:41.013276+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:41.013276+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:41.013277+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:41.015082+00:00 app[web.1]: E, [2016-09-16T17:40:40.991259 #14006] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:41.015083+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:41.015084+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:41.015085+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:41.015086+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:41.015087+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:41.015088+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:41.015086+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:41.015088+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:41.015089+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:41.029456+00:00 app[web.1]: E, [2016-09-16T17:40:41.019600 #14013] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:41.029459+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:41.029461+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:41.029462+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:41.029461+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:41.029462+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:41.029464+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:41.029464+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:41.029465+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:41.029466+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:41.033517+00:00 app[web.1]: E, [2016-09-16T17:40:41.032905 #3] ERROR -- : reaped #<Process::Status: pid 14006 exit 1> worker=1
2016-09-16T17:40:41.033606+00:00 app[web.1]: E, [2016-09-16T17:40:41.033566 #3] ERROR -- : reaped #<Process::Status: pid 14009 exit 1> worker=2
2016-09-16T17:40:41.034410+00:00 app[web.1]: E, [2016-09-16T17:40:41.034370 #3] ERROR -- : reaped #<Process::Status: pid 14013 exit 1> worker=0
2016-09-16T17:40:41.067342+00:00 app[web.1]: E, [2016-09-16T17:40:41.057962 #14018] ERROR -- : uninitialized constant #<Class:#<Unicorn::Configurator:0x007f7b03390af0>>::Sidekiq (NameError)
2016-09-16T17:40:41.067346+00:00 app[web.1]: ./config/unicorn.rb:7:in `block in reload'
2016-09-16T17:40:41.067347+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:611:in `init_worker_process'
2016-09-16T17:40:41.067347+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:639:in `worker_loop'
2016-09-16T17:40:41.067348+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
2016-09-16T17:40:41.067349+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
2016-09-16T17:40:41.067349+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
2016-09-16T17:40:41.067350+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
2016-09-16T17:40:41.067351+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
2016-09-16T17:40:41.067352+00:00 app[web.1]: /app/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
2016-09-16T17:40:41.077263+00:00 app[web.1]: E, [2016-09-16T17:40:41.077156 #3] ERROR -- : reaped #<Process::Status: pid 14018 exit 1> worker=1
Here is my unicorn config
before_fork do |server, worker|
@sidekiq_pid ||= spawn("bundle exec sidekiq -c 2")
end
worker_processes 3
after_fork do |server, worker|
Sidekiq.configure_client do |config|
config.redis = { :size => 1 }
end
Sidekiq.configure_server do |config|
config.redis = { :size => 5 }
end
end
web: bundle exec unicorn -p $PORT -E $RACK_ENV -c ./config/unicorn.rb
Firstly, setup your procfile to properly utilize the worker dynos (im using puma for web so change that to reflect your unicorn setup):
web: bundle exec puma -C config/puma.rb
worker: bundle exec rake jobs:work
Now, to create a rake task is very simple. Open up your lib/tasks folder and add a file called "scraper.rake". This is where you're going to put the code that runs your scraper. Since you didn't show your scraper code I added some of mine so you have an example. First, require any outside files or configs youll need for running the scraper. Then you wrap everything inside a namespace (I chose :daily) but you can pick anything to help make it more descriptive. Then, the task line should be completely copied. The ":environment do" part just tells rails to utilize models and the env. while running rake.
require 'capybara/poltergeist'
namespace :daily do
desc "Scrapes hourly for much needed dataaaaa"
task :scrape => :environment do
Capybara.register_driver(:poltergeist) { |app| Capybara::Poltergeist::Driver.new(app, {js_errors: true, inspector: true, debug: false})} #, extensions: ["vendor/poltergeist/phantom_config.js"]}) }
Capybara.default_max_wait_time = 12
Capybara.default_driver = :poltergeist # configure Capybara to use poltergeist as the driver
Capybara.javascript_driver = :poltergeist
Capybara.current_driver = :poltergeist
@session = Capybara.current_session
EnvisionAccount.find_each do |envision_account|
# account = Scraper::HourlyScraper.new
daily_sign_in(envision_account)
end
end
end
Once your code is setup like this you can run your scraper from the command line by doing: bundle exec rake daily:scrape If everything works you just need to go onto Heroku, add the scheduler Addon and then tell it to run that rake command on whatever interval you need. Also note, Heroku needs a buildpack if you're using capybara and poltergeist for scraping so run this:
heroku buildpacks:add --index 1 https://github.com/stomita/heroku-buildpack-phantomjs
Enjoy