I am attempting to migrate from Heroku to AWS, but my Sidekiq jobs keep failing with the following error:
Errno::EPIPE: Broken pipe @ io_write - <STDOUT>
I can successfully run jobs from the console using perform_now, and everything works just fine in Heroku, so I am presuming the issue lies somewhere with my AWS setup. I have seen references to improper daemonization around Stack Overflow and Github but not sure how to solve the problem.
Right now I am launching my processes with the following command:
foreman start -f Procfile -p 3000 -e $VAR_FILES &
and I have tried the command both with and without the & at the end.
My Procfile looks like this:
web: bundle exec puma -t 1:2 -p ${PORT:-3000} -e ${RACK_ENV:-production}
worker: bundle exec sidekiq -C config/sidekiq.yml
log: tail -f log/production.log
and I have also tried it like this, following the instructions here (https://github.com/mperham/sidekiq/wiki/Logging#syslog):
worker: bundle exec sidekiq -C config/sidekiq.yml 2>&1 | logger -t sidekiq
My sidekiq.yml has logfile set to ./log/sidekiq.log, which I believe is supposed to redirecting logs away from STDOUT anyway.
I have seen the discussion here (https://github.com/mperham/sidekiq/issues/3188) and can verify that the rails12factor gem is not in my Gemfile.
But still the error persists... Can anyone lend a hand?
UPDATE: I can finally get a stack trace and see it is coming from a puts statement inside of the Neo4j.rb gem:
2017-04-07T15:46:53.553Z 697 TID-12a6r4 WARN: Errno::EPIPE: Broken pipe @ io_write - <STDOUT>
2017-04-07T15:46:53.553Z 697 TID-12a6r4 WARN: /var/lib/gems/2.3.0/bundler/gems/neo4j-c804cb33bef8/lib/neo4j/session_manager.rb:60:in `write'
/var/lib/gems/2.3.0/bundler/gems/neo4j-c804cb33bef8/lib/neo4j/session_manager.rb:60:in `puts'
/var/lib/gems/2.3.0/bundler/gems/neo4j-c804cb33bef8/lib/neo4j/session_manager.rb:60:in `puts'
But still not sure how I can mitigate the issue. I have tried with RAILS_LOG_TO_STDOUT=enabled both set and unset.
I spoke to the gem maintainers and they removed the puts statements in v 8.0.13. It fixed the problem for me!