My Rails 3.2.21 app was running fine on AWS Beanstalk under Passenger Standalone 4.0.53. I migrated the app to Rails 4.2.1 and got it passing all tests on my local development machine (Ubuntu, WEBrick). I deployed it to Beanstalk (aws.push), the deploy succeeds (copied from /ondeck to /current) and: nothing. I browse to the site and see a blank page. No 404 error, nothing.
On the old version, I had pre-compiled assets locally. This time, I let Beanstalk run 'bundle exec rake assets:precompile' webapp. I see in eb-activity.log that this succeeded, and I see those assets in var/app/current/public/assets. But this isn't about not serving public assets, it's about not serving anything.
So yes, there is a /public directory as required by Passenger. There is also a /tmp directory. And here's the config.ru that Rails created:
require ::File.expand_path('../config/environment', __FILE__)
run Rails.application
Also in eb-activity.log, I see this message that seems to indicate Passenger is working:
+ service passenger restart
=============== Phusion Passenger Standalone web server started ===============
PID file: /var/app/support/pids/passenger.pid
Log file: /var/app/support/logs/passenger.log
Environment: production
Accessible via: http://0.0.0.0/
Serving in the background as a daemon.
But when I run curl -o - http://0.0.0.0 from the console, it just goes to the next line--no headers or other web content is returned.
I can run rails c to start a Rails console on the AWS machine. Works fine, has database access.
I've re-run eb init on my development machine to make sure it connects correctly, although the git aws.push has always worked to upload a new version.
I terminated the EC2 instance and let the load balancer start a new instance. That built successfully. The first time it came up (and only then), I saw some error messages in /var/app/support/logs/passenger.log:
[ 2015-06-17 01:03:54.8281 2556/7fd8941b8740 agents/Watchdog/Main.cpp:538 ]: Options: { [redacted] }
[ 2015-06-17 01:03:55.2388 2559/7ff254abe740 agents/HelperAgent/Main.cpp:650 ]: PassengerHelperAgent online, listening at unix:/tmp/passenger.1.0.2555/generation-0/request
[ 2015-06-17 01:03:55.9207 2567/7fcb54570740 agents/LoggingAgent/Main.cpp:321 ]: PassengerLoggingAgent online, listening at unix:/tmp/passenger.1.0.2555/generation-0/logging
[ 2015-06-17 01:03:55.9209 2556/7fd8941b8740 agents/Watchdog/Main.cpp:728 ]: All Phusion Passenger agents started!
2015/06/17 01:03:57 [error] 2575#0: *3 "/var/app/current/public/index.html" is not found (2: No such file or directory), client: 127.0.0.1, server: _, request: "HEAD / HTTP/1.1", host: "0.0.0.0"
2015/06/17 01:04:02 [error] 2575#0: *4 "/var/app/current/public/app_check/index.html" is not found (2: No such file or directory), client: 172.31.21.97, server: _, request: "GET /app_check/ HTTP/1.1", host: "172.31.25.47"
[ 2015-06-17 01:06:10.4000 21317/7f433c999740 agents/Watchdog/Main.cpp:538 ]: Options: { [redacted] }
[ 2015-06-17 01:06:10.4047 21320/7f414cdd5740 agents/HelperAgent/Main.cpp:650 ]: PassengerHelperAgent online, listening at unix:/tmp/passenger.1.0.21316/generation-0/request
[ 2015-06-17 01:06:10.4090 21325/7f87c44f2740 agents/LoggingAgent/Main.cpp:321 ]: PassengerLoggingAgent online, listening at unix:/tmp/passenger.1.0.21316/generation-0/logging
[ 2015-06-17 01:06:10.4092 21317/7f433c999740 agents/Watchdog/Main.cpp:728 ]: All Phusion Passenger agents started!
Usually, passenger.log is empty, even after I try to access the site.
I just tried setting the log level to DEBUG. After Passenger restarted, it printed this in passenger.log:
[ 2015-06-18 17:52:28.3921 631/7f42fc9ce700 Pool2/SmartSpawner.h:298 ]: Preloader for /var/app/current started on PID 666, listening on unix:/tmp/passenger.1.0.627/generation-0/backends/preloader.p23dhh
Why isn't Passenger serving web pages? How can I troubleshoot this?
Update 6/18/2015
Tried pre-compiling assets on development machine and then deploying. Didn't help.
Update 6/19/2015
I now have a Rails 4.2.1 test app successfully running under the same Beanstalk environment (64bit Amazon Linux 2014.09 v1.1.0 running Ruby 2.0 - Passenger Standalone). I see that on the test app, there are four Passenger processes:
The pid returned by service passenger status corresponds to the PassengerHelper running as root.
On the failing live app, there is only one Passenger process:
The pid returned by service passenger status does not correspond to an active process.
So apparently, three of the four Passenger processes are crashing after they start. So far I have not been able to find any corresponding error logs, so I don't know why they crash.
I finally figured out how to increase the logging level for Passenger Standalone (blogged here). From the log, I could see that the web server was responding to the Beanstalk health check with 301 redirects. That meant that the load balancer thought the app was dead, so it was sending 503 errors back to the browser, which displayed a blank page.
The 301 redirect tipped me to the force_ssl configuration. I had it configured as follows in production.rb:
config.force_ssl = true
config.ssl_options = { exclude: proc { |env| env['PATH_INFO'].start_with?('/app_check') } }
The second line, following this post, is supposed to allow the health check (which only works on http) to bypass the force_ssl> However, the exclude option has been removed from Rail 4.2 per this commit. Without the exclude, the health check failed and the load balancer wouldn't serve the app.
The default Elastic Beanstalk behavior, if you don't specify a health check URL, is to simply ping the server on port 22. However I prefer that the health check actually confirm that the web server is working by loading a web page.
I've implemented this workaround:
Remove config.force_ssl and config.ssl_options from production.rb.
Add the following to application.rb, assuming that the health check is accessible from StaticPagesController#health_check:
force_ssl if: :require_ssl? # see private method below for conditions
private
def require_ssl?
Rails.env.production? && !(controller_name == "static_pages" && action_name == "health_check")
end
I may still have a Passenger or Nginx issue since I still only have one Passenger process running, and its PID doesn't match the PID from service passenger status. But at least the web server is serving my site again.