I'm using Foreman and Capistrano to deploy my rails app. The web (i.e. Puma) process starts correctly, but I can't get my worker (i.e sidekiq) to run as a production environment.
I adapted the app-worker-1.service file generated in the SystemD export while tying to force the environment to be set correctly, without success:
[Unit]
StopWhenUnneeded=true
[Service]
User=deployment_user
WorkingDirectory=/var/www/app_name/current
ExecStart=/bin/bash -lc "RAILS_ENV=production bundle exec sidekiq -e production"
RestartSec=1
Restart=on-failure
StandardOutput=journal
StandardError=journal
SyslogIdentifier=%n
KillMode=process
I tried everything...I added Logger.debug statements that shows the RACK_ENV and RAILS_ENV are both incorrectly set to development.
Here is the error I get from the process journal:
I, [2015-09-19T13:56:14.813528 #31187] INFO -- : Celluloid 0.17.1.2 is running in BACKPORTED mode. [ http://git.io/vJf3J ]
D, [2015-09-19T13:56:16.765338 #31187] DEBUG -- : RACK_ENV=development
D, [2015-09-19T13:56:16.766057 #31187] DEBUG -- : RAILS_ENV=development
'development' database is not configured. Available: ["production"]
/var/www/app_name/shared/bundle/ruby/2.2.0/gems/activerecord-4.2.4/lib/active_record/connection_adapters/connection_specification.rb:248:in `resolve_symbol_connection'
/var/www/app_name/shared/bundle/ruby/2.2.0/gems/activerecord-4.2.4/lib/active_record/connection_adapters/connection_specification.rb:211:in `resolve_connection'
/var/www/app_name/shared/bundle/ruby/2.2.0/gems/activerecord-4.2.4/lib/active_record/connection_adapters/connection_specification.rb:139:in `resolve'
/var/www/app_name/shared/bundle/ruby/2.2.0/gems/activerecord-4.2.4/lib/active_record/connection_adapters/connection_specification.rb:169:in `spec'
/var/www/app_name/shared/bundle/ruby/2.2.0/gems/activerecord-4.2.4/lib/active_record/connection_handling.rb:50:in `establish_connection'
/var/www/app_name/shared/bundle/ruby/2.2.0/gems/activerecord-4.2.4/lib/active_record/railtie.rb:120:in `block (2 levels) in <class:Railtie>'
/var/www/app_name/shared/bundle/ruby/2.2.0/gems/activesupport-4.2.4/lib/active_support/lazy_load_hooks.rb:38:in `instance_eval'
/var/www/app_name/shared/bundle/ruby/2.2.0/gems/activesupport-4.2.4/lib/active_support/lazy_load_hooks.rb:38:in `execute_hook'
/var/www/app_name/shared/bundle/ruby/2.2.0/gems/activesupport-4.2.4/lib/active_support/lazy_load_hooks.rb:28:in `block in on_load'
/var/www/app_name/shared/bundle/ruby/2.2.0/gems/activesupport-4.2.4/lib/active_support/lazy_load_hooks.rb:27:in `each'
/var/www/app_name/shared/bundle/ruby/2.2.0/gems/activesupport-4.2.4/lib/active_support/lazy_load_hooks.rb:27:in `on_load'
/var/www/app_name/shared/bundle/ruby/2.2.0/gems/activerecord-4.2.4/lib/active_record/railtie.rb:116:in `block in <class:Railtie>'
/var/www/app_name/shared/bundle/ruby/2.2.0/gems/railties-4.2.4/lib/rails/initializable.rb:30:in `instance_exec'
/var/www/app_name/shared/bundle/ruby/2.2.0/gems/railties-4.2.4/lib/rails/initializable.rb:30:in `run'
/var/www/app_name/shared/bundle/ruby/2.2.0/gems/railties-4.2.4/lib/rails/initializable.rb:55:in `block in run_initializers'
/usr/local/rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/tsort.rb:226:in `block in tsort_each'
/usr/local/rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/tsort.rb:348:in `block (2 levels) in each_strongly_connected_component'
/usr/local/rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/tsort.rb:429:in `each_strongly_connected_component_from'
/usr/local/rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/tsort.rb:347:in `block in each_strongly_connected_component'
/usr/local/rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/tsort.rb:345:in `each'
/usr/local/rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/tsort.rb:345:in `call'
/usr/local/rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/tsort.rb:345:in `each_strongly_connected_component'
/usr/local/rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/tsort.rb:224:in `tsort_each'
/usr/local/rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/tsort.rb:203:in `tsort_each'
/var/www/app_name/shared/bundle/ruby/2.2.0/gems/railties-4.2.4/lib/rails/initializable.rb:54:in `run_initializers'
/var/www/app_name/shared/bundle/ruby/2.2.0/gems/railties-4.2.4/lib/rails/application.rb:352:in `initialize!'
/var/www/app_name/shared/bundle/ruby/2.2.0/gems/railties-4.2.4/lib/rails/railtie.rb:194:in `public_send'
/var/www/app_name/shared/bundle/ruby/2.2.0/gems/railties-4.2.4/lib/rails/railtie.rb:194:in `method_missing'
/var/www/app_name/releases/20150918131757/config/environment.rb:12:in `<top (required)>'
...
Found it!
First I enabled debugging on SystemD:
Environment=SYSTEMD_LOG_LEVEL=debug
Then while looking a my services journal, my attention was drawn to a message that looked unrelated at first prompting to look at the service status itself...
capistrano@droplet2:/lib/systemd/system$ systemctl status app_name-worker-1.service
● alagarderie-web-1.service
Loaded: error (Reason: Invalid argument)
Active: inactive (dead) since Sat 2015-09-19 20:25:14 EDT; 35min ago
Main PID: 32292 (code=exited, status=0/SUCCESS)
...
... the first line are not important ...
...
Warning: alagarderie-web-1.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Warning, WHAT! I knew I'd hate myself when I find the problem...
Changes to a SystemD script file are not synced automatically. I reloaded my app target, then started it; et voila!