Search code examples
ruby-on-railsredissidekiq

Sidekiq/Redis/Rails - Sending was working on the branch, but now will not deliver on development

Ruby version: 2.6.5 Rails version: 6.1.3.1 Sidekiq / Pro / Enterprise version(s): 6.2

Initializer: No initializer

Sidekiq.yml:

:verbose: true
:concurrency: 25
:queues:
 - [mailers, 7]
 - [default, 5]

Error Messages: NONE

Full Sidekiq Backtrace:

****** (development) $ bundle exec sidekiq


               m,
               `$b
          .ss,  $$:         .,d$
          `$$P,d$P'    .,md$P"'
           ,$$$$$b/md$$$P^'
         .d$$$$$$/$$$P'
         $$^' `"/$$$'       ____  _     _      _    _
         $:     ,$$:       / ___|(_) __| | ___| | _(_) __ _
         `b     :$$        \___ \| |/ _` |/ _ \ |/ / |/ _` |
                $$:         ___) | | (_| |  __/   <| | (_| |
                $$         |____/|_|\__,_|\___|_|\_\_|\__, |
              .d$$                                       |_|


2021-06-13T15:22:14.594Z pid=756 tid=oxyx50z9s INFO: Booted Rails 6.1.3.1 application in development environment
2021-06-13T15:22:14.594Z pid=756 tid=oxyx50z9s INFO: Running in ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]
2021-06-13T15:22:14.594Z pid=756 tid=oxyx50z9s INFO: See LICENSE and the LGPL-3.0 for licensing details.
2021-06-13T15:22:14.595Z pid=756 tid=oxyx50z9s INFO: Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org
2021-06-13T15:22:14.595Z pid=756 tid=oxyx50z9s INFO: Booting Sidekiq 6.2.1 with redis options {}
2021-06-13T15:22:14.597Z pid=756 tid=oxyx50z9s DEBUG: Client Middleware:
2021-06-13T15:22:14.597Z pid=756 tid=oxyx50z9s DEBUG: Server Middleware: Raven::Sidekiq::CleanupMiddleware
2021-06-13T15:22:14.598Z pid=756 tid=oxyx50z9s INFO: Starting processing, hit Ctrl-C to stop
2021-06-13T15:22:14.598Z pid=756 tid=oxyx50z9s DEBUG: {:queues=>["mailers", "mailers", "mailers", "mailers", "mailers", "mailers", "mailers", "default", "default", "default", "default", "default"], :labels=>[], :concurrency=>25, :require=>".", :strict=>false, :environment=>nil, :timeout=>25, :poll_interval_average=>nil, :average_scheduled_poll_interval=>5, :error_handlers=>[#<Sidekiq::ExceptionHandler::Logger:0x00007ffff8446a88>, #<Raven::Sidekiq::ErrorHandler:0x00007ffffa9bddc0>], :death_handlers=>[], :lifecycle_events=>{:startup=>[], :quiet=>[], :shutdown=>[], :heartbeat=>[]}, :dead_max_jobs=>10000, :dead_timeout_in_seconds=>15552000, :reloader=>#<Sidekiq::Rails::Reloader @app=Bdcommunity::Application>, :verbose=>true, :config_file=>"./config/sidekiq.yml", :tag=>"bdcommunity", :identity=>"DESKTOP***", :fetch=>#<Sidekiq::BasicFetch:0x00007ffffd789bc8 @options={...}, @strictly_ordered_queues=false, @queues=["queue:mailers", "queue:mailers", "queue:mailers", "queue:mailers", "queue:mailers", "queue:mailers", "queue:mailers", "queue:default", "queue:default", "queue:default", "queue:default", "queue:default"]>}
2021-06-13T15:23:41.567Z pid=756 tid=oxyy3dekc class=ActionMailer::MailDeliveryJob jid=1430f4cf1b9ea1c8f7578578 INFO: start
2021-06-13T15:23:41.584Z pid=756 tid=oxyy3dfsg class=ActionMailer::MailDeliveryJob jid=24dc64f44ab2088912a506e2 INFO: start
2021-06-13T15:23:41.584Z pid=756 tid=oxyy0umak class=ActionMailer::MailDeliveryJob jid=dce3868b2802b2a44c964f80 INFO: start
2021-06-13T15:23:41.584Z pid=756 tid=oxyy3dgjc class=ActionMailer::MailDeliveryJob jid=c85bd721343289b44218bd5d INFO: start
2021-06-13T15:23:41.585Z pid=756 tid=oxyy3df2o class=ActionMailer::MailDeliveryJob jid=81d4a665cc70e7056bfee2bc INFO: start

Full Redis Backtrace:

630:M 13 Jun 2021 11:21:44.283 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
630:M 13 Jun 2021 11:21:44.283 # Server initialized
630:M 13 Jun 2021 11:21:44.283 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
630:M 13 Jun 2021 11:21:44.283 * Loading RDB produced by version 6.2.1
630:M 13 Jun 2021 11:21:44.283 * RDB age 152 seconds
630:M 13 Jun 2021 11:21:44.283 * RDB memory usage when created 1.42 Mb
630:M 13 Jun 2021 11:21:44.284 * DB loaded from disk: 0.000 seconds
630:M 13 Jun 2021 11:21:44.284 * Ready to accept connections
630:M 13 Jun 2021 11:26:45.030 * 100 changes in 300 seconds. Saving...
630:M 13 Jun 2021 11:26:45.037 * Background saving started by pid 1004
1004:C 13 Jun 2021 11:26:45.050 * DB saved on disk
630:M 13 Jun 2021 11:26:45.138 * Background saving terminated with success
630:M 13 Jun 2021 11:31:46.019 * 100 changes in 300 seconds. Saving...
630:M 13 Jun 2021 11:31:46.026 * Background saving started by pid 1216
1216:C 13 Jun 2021 11:31:46.047 * DB saved on disk
630:M 13 Jun 2021 11:31:46.127 * Background saving terminated with success

Full Relevant Rails Server Backtrace:

[ActiveJob] Enqueued ActionMailer::MailDeliveryJob (Job ID: 7d4823ed-e0f8-40ec-9a25-ddac8e373de5) to Sidekiq(mailers) with arguments: "NotificationMailer", "private_message_received", "deliver_now", {:args=>[#<GlobalID:0x00007fffd89c8008 @uri=#<URI::GID gid://***>>, #<GlobalID:0x00007fffd89ff918 @uri=#<URI::GID gid://***>>, "Test Number 3"]}
[ActiveJob] Enqueued ActionMailer::MailDeliveryJob (Job ID: ac28e08d-3318-41e5-83c8-83010be6a1d3) to Sidekiq(mailers) with arguments: "NotificationMailer", "private_message_received", "deliver_now", {:args=>[#<GlobalID:0x00007fffd8bbbb80 @uri=#<URI::GID gid://***>>, #<GlobalID:0x00007fffd8c006b8 @uri=#<URI::GID gid://***>>, "Test Number 3"]}
[ActiveJob] Enqueued ActionMailer::MailDeliveryJob (Job ID: 87e09cd3-631e-46c3-9f2e-0e7f622d2d7a) to Sidekiq(mailers) with arguments: "NotificationMailer", "private_message_received", "deliver_now", {:args=>[#<GlobalID:0x00007fffd8ef99f0 @uri=#<URI::GID gid://***>>, #<GlobalID:0x00007fffd8f35770 @uri=#<URI::GID gid://***>>, "Test Number 3"]}
[ActiveJob] Enqueued ActionMailer::MailDeliveryJob (Job ID: 3eebfc8c-faba-40d0-9edf-64e34cd1161b) to Sidekiq(mailers) with arguments: "NotificationMailer", "private_message_received", "deliver_now", {:args=>[#<GlobalID:0x00007fffd91572d8 @uri=#<URI::GID gid://***>>, #<GlobalID:0x00007fffd91796f8 @uri=#<URI::GID gid://***>>, "Test Number 3"]}
[ActiveJob] Enqueued ActionMailer::MailDeliveryJob (Job ID: 26cca661-437f-4d3e-a341-02fbe53ebb6a) to Sidekiq(mailers) with arguments: "NotificationMailer", "private_message_received", "deliver_now", {:args=>[#<GlobalID:0x00007fffd934e2d0 @uri=#<URI::GID gid://***/***/**>>, #<GlobalID:0x00007fffd9352628 @uri=#<URI::GID gid://****>>, "Test Number 3"]}
Completed 302 Found in 439ms (ActiveRecord: 25.8ms | Allocations: 80420)

Does anyone know why the emails are now not delivering?

Solution

  • The answer turned out to be that production needs a LOT of settings to get sidekiq to work.

    The first main change was adding a sidekiq.rb initializer:

    Sidekiq.default_worker_options = {
  backtrace: true,
  retry: true
}

sidekiq_redis = lambda do
  Redis.new(url: ENV['REDIS_URL'], ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE })
end

Sidekiq.configure_client do |config|
  config.redis = ConnectionPool.new(size: 2, &sidekiq_redis)
end

Sidekiq.configure_server do |config|
  config.redis = ConnectionPool.new(size: 17, &sidekiq_redis)
end

Sidekiq::Extensions.enable_delay!

    Note that some of that (the OpenSSL part) is from a NEW issue we had to solve, when we upgraded from the free heroku redis plan to the lowest paid tier. TIP: Your app will crash if you upgrade to the first paid tier without making a bunch of adjustments. So do a lot of research before you do leave the free heroku redis plan.

    We also had to add a new WORKER. We originally just had one regular dyno. We had to actually add a worker, like this:

    Image from Heroku Showing the Worker We Added

    Notice that there's still a sidekiq.yml. You need that too:

    :verbose: true
:concurrency: 15
:queues:
 - [mailers, 7]
 - [default, 5]
 - List item

    Further, if you use actioncable, you'll likely have to update the actioncable.yml file as well:

    development:
  adapter: redis
  url: redis://localhost:6379/1

test:
  adapter: async

production:
  adapter: redis
  url: <%= ENV['REDIS_URL'] %>
  ssl_params:
    verify_mode: <%= OpenSSL::SSL::VERIFY_NONE %>

    Again, that last part, with the ssl_params is to fix the issues caused when upgrading to the paid tier of redis.

    NOTE: Although our emails deliver now, we haven't yet fully solved the SSL issue. We have a support ticket in with Heroku. But I'm hoping at least this will help you to better see what you may need to change to get sidekiq to work initially.

    One final note: we learned that RedisToGo only works with Redis 4 or something. So if you want to use sidekiq, you can't use RedisToGo, because it doesn't apparently work with Redis 4. You need to use Heroku Redis. That was another change we made, which allowed sidekiq to work on production.