We're using rest-client to send external API requests on Sidekiq workers that spin up as a process on our Elastic Beanstalk-managed EC2 instances.
Under the hood rest-client uses netrc, and netrc calls Dir.pwd when Dir.home fails. We are seeing this error sporadically:
Errno::ENOENT: No such file or directory - getcwd
[GEM_ROOT]/gems/netrc-0.10.3/lib/netrc.rb:26 :in `pwd`
Which indicates that ENV['HOME'] is not set, and apparently the current directory no longer exists. Generally HOME is set already on EB instances, and we've attempted to set it via eb setenv just to be sure.
The question is how this situation might arise on EB and how we can avoid it in the future. I hesitate to speculate but perhaps we are not killing Sidekiq processes correctly between deployments and old sidekiq processes survive, referencing old directories? I'm not terribly familiar with EB yet; any guidance would be much appreciated.
As foreshadowed in both the question and some answers this was a rogue process that avoided termination, in this case due to a careless config change in one of our early deployments. Line 30 of our sidekiq.config controls the number of processes, and we dropped from 2 to 1 but did not manually remove the second process from the server after the change.
The moral of the story is that Sidekiq processes are not only vulnerable to unexpected behavior during deployments. Our errant process was happily completing jobs even without being stopped/started across multiple deployments, but anything that relied on Dir.pwd and presumably any other script requiring ENV data would fail days after a stable deployment if run on the persistent process.
If you see this type of failure, be absolutely positive you don't have unexpected processes running from prior deployments. This means ensuring they are properly terminated during deployment as @JeffD23 mentioned but ALSO ensuring you don't have something hanging around from deployments of yore.