I have a Rails 6 website running on Elastic Beanstalk (Amazon Linux 2). I successfully implemented a process to use Certbot to generate an SSL certificate, and when I visit my website everything is working correctly. However, when I try to log in to my user console (using Devise), I receive 422 errors.
Rails Production Log
W, [2021-11-26T17:55:17.528942 #22645] WARN -- : [6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] HTTP Origin header (https://example.com) didn't match request.base_url (http://example.com)
I, [2021-11-26T17:55:17.529316 #22645] INFO -- : [6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] Completed 422 Unprocessable Entity in 1ms (ActiveRecord: 0.0ms | Allocations: 565)
F, [2021-11-26T17:55:17.530195 #22645] FATAL -- : [6ef6bfd6-6d78-4ded-90df-a9472e0d40f6]
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] ActionController::InvalidAuthenticityToken (ActionController::InvalidAuthenticityToken):
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6]
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_controller/metal/request_forgery_protection.rb:211:in `handle_unverified_request'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_controller/metal/request_forgery_protection.rb:243:in `handle_unverified_request'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] devise (4.8.0) lib/devise/controllers/helpers.rb:255:in `handle_unverified_request'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_controller/metal/request_forgery_protection.rb:238:in `verify_authenticity_token'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/callbacks.rb:427:in `block in make_lambda'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/abstract_controller/callbacks.rb:34:in `block (2 levels) in <module:Callbacks>'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/callbacks.rb:199:in `block in halting'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/callbacks.rb:512:in `block in invoke_before'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/callbacks.rb:512:in `each'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/callbacks.rb:512:in `invoke_before'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/callbacks.rb:115:in `block in run_callbacks'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actiontext (6.1.4.1) lib/action_text/rendering.rb:20:in `with_renderer'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actiontext (6.1.4.1) lib/action_text/engine.rb:59:in `block (4 levels) in <class:Engine>'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/callbacks.rb:126:in `instance_exec'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/callbacks.rb:126:in `block in run_callbacks'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/callbacks.rb:137:in `run_callbacks'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/abstract_controller/callbacks.rb:41:in `process_action'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_controller/metal/rescue.rb:22:in `process_action'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/notifications.rb:203:in `block in instrument'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/notifications/instrumenter.rb:24:in `instrument'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/notifications.rb:203:in `instrument'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_controller/metal/instrumentation.rb:33:in `process_action'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activerecord (6.1.4.1) lib/active_record/railties/controller_runtime.rb:27:in `process_action'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/abstract_controller/base.rb:165:in `process'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionview (6.1.4.1) lib/action_view/rendering.rb:39:in `process'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_controller/metal.rb:190:in `dispatch'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_controller/metal.rb:254:in `dispatch'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/routing/route_set.rb:33:in `serve'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/routing/mapper.rb:49:in `serve'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/journey/router.rb:50:in `block in serve'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/journey/router.rb:32:in `each'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/journey/router.rb:32:in `serve'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/routing/route_set.rb:842:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] warden (1.2.9) lib/warden/manager.rb:36:in `block in call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] warden (1.2.9) lib/warden/manager.rb:34:in `catch'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] warden (1.2.9) lib/warden/manager.rb:34:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] rack (2.2.3) lib/rack/tempfile_reaper.rb:15:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] rack (2.2.3) lib/rack/etag.rb:27:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] rack (2.2.3) lib/rack/conditional_get.rb:40:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] rack (2.2.3) lib/rack/head.rb:12:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/http/permissions_policy.rb:22:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/http/content_security_policy.rb:18:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] rack (2.2.3) lib/rack/session/abstract/id.rb:266:in `context'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] rack (2.2.3) lib/rack/session/abstract/id.rb:260:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/middleware/cookies.rb:689:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/callbacks.rb:98:in `run_callbacks'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/middleware/callbacks.rb:26:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] railties (6.1.4.1) lib/rails/rack/logger.rb:37:in `call_app'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] railties (6.1.4.1) lib/rails/rack/logger.rb:26:in `block in call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/tagged_logging.rb:99:in `block in tagged'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/tagged_logging.rb:37:in `tagged'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/tagged_logging.rb:99:in `tagged'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] railties (6.1.4.1) lib/rails/rack/logger.rb:26:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/middleware/request_id.rb:26:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] rack (2.2.3) lib/rack/method_override.rb:24:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] rack (2.2.3) lib/rack/runtime.rb:22:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] activesupport (6.1.4.1) lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/middleware/executor.rb:14:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] rack (2.2.3) lib/rack/sendfile.rb:110:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] actionpack (6.1.4.1) lib/action_dispatch/middleware/host_authorization.rb:92:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] railties (6.1.4.1) lib/rails/engine.rb:539:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] puma (5.5.2) lib/puma/configuration.rb:249:in `call'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] puma (5.5.2) lib/puma/request.rb:77:in `block in handle_request'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] puma (5.5.2) lib/puma/thread_pool.rb:340:in `with_force_shutdown'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] puma (5.5.2) lib/puma/request.rb:76:in `handle_request'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] puma (5.5.2) lib/puma/server.rb:447:in `process_client'
[6ef6bfd6-6d78-4ded-90df-a9472e0d40f6] puma (5.5.2) lib/puma/thread_pool.rb:147:in `block in spawn_thread'
Puma Log (most recent lines, nothing relevant in here as far as I can tell)
[22565] - Worker 0 (PID: 22643) booted in 5.67s, phase: 0
[22565] - Worker 1 (PID: 22645) booted in 5.68s, phase: 0
Nginx Access Log
99.83.42.176 - - [26/Nov/2021:17:55:10 +0000] "GET /users/sign_in HTTP/1.1" 200 7754 "https://example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.55 Safari/537.36" "-"
99.83.42.176 - - [26/Nov/2021:17:55:17 +0000] "POST /users/sign_in HTTP/1.1" 422 0 "https://example.com/users/sign_in" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.55 Safari/537.36" "-"
Nginx Error Log
2021/11/26 17:54:57 [notice] 22658#22658: signal process started
2021/11/26 17:54:57 [warn] 22626#22626: conflicting server name "localhost" on 0.0.0.0:80, ignored
2021/11/26 17:54:57 [warn] 22626#22626: conflicting server name "_" on 0.0.0.0:80, ignored
I've messed with a few options in nginx.conf but still haven't been able to track down why this is happening. This is my nginx.conf file, with the certbot-embedded changes:
#Elastic Beanstalk Nginx Configuration File
user nginx;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
worker_processes auto;
worker_rlimit_nofile 65874;
events {
worker_connections 1024;
}
http {
upstream appserver {
server unix:/var/run/puma/my_app.sock;
}
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
include conf.d/*.conf;
map $http_upgrade $connection_upgrade {
default "upgrade";
}
server {
listen 80;
server_name www.example.com example.com localhost _;
return 301 https://$host$request_uri;
}
server {
client_header_timeout 60;
client_body_timeout 60;
keepalive_timeout 60;
gzip on;
gzip_comp_level 4;
gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript;
# Include the Elastic Beanstalk generated locations
include conf.d/elasticbeanstalk/*.conf;
server_name www.example.com example.com localhost _; # managed by Certbot
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
location appserver {
proxy_http_version 1.1;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header X-Forwarded-Proto https;
proxy_pass http://appserver;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
}
}
server {
if ($host = www.example.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
if ($host = example.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80 ;
server_name www.example.com example.com localhost _;
return 404; # managed by Certbot
}
}
Has anyone faced this problem, and know how to make sure nginx handles everything as https?
Not sure if this in related, but in the production config I have:
config.force_ssl = false
If I switch it to 'true', nothing loads at all. Not sure why, shouldn't that be 'true'?
@jamesc's suggestion ultimately led me to a solution. Elastic Beanstalk's AL2 Nginx confix file includes a webapp.conf file with the following block:
location @proxy {
proxy_pass http://my_app; # match the name of upstream directive which is defined above
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
That wasn't passing through the full set of headers, and was resulting in invalid requests. I updated my nginx.conf file to provide an updated location block based on this thread: https://github.com/rails/rails/issues/22965.
location / {
proxy_pass http://my_app;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Ssl on;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Host $host;
}