Sidekiq Redis database keys increasing over time

Question

I am currently using Sidekiq with my Rails app in production along with an ElasticCache Redis database. I've noticed recently that when monitoring the CurrItems metric using the AWS tools, I see the number of items gradually increasing over time in an almost step-like way:

However, when I look at the jobs in queue in the Sidekiq dashboard, I don't see anything backing up at all. I see 0 jobs in queue, 0 busy, 0 scheduled.

The step-like increase seems to happen at a very particular time each day (right at the end of the day), which made me think it might be related to a chron job/clockwork process I have running. However, I only have 4 jobs that run once a day and none of them run during that time or even near that time. Just for good measure though, here is my clock.rb file (I have shorted all the job descriptions and class and method names for simplicity's sake):

module Clockwork
  every(30.seconds, 'Task 1') { Class.method }
  every(30.seconds, 'Task 2') { Class.method }
  every(10.minutes, 'Task 3') { Class.method }
  every(1.day, 'Task 4', :at => '06:00', :tz => 'EST') { Class.method }
  every(10.minutes, 'Task 5') { Class.method }
  every(1.day, 'Task 6', :at => '20:00', :tz => 'UTC') { Class.method }
  every(1.day, 'Task 7', :at => '20:00', :tz => 'UTC') { Class.method }
  every(1.day, 'Task 8', :at => '20:00', :tz => 'UTC') { Class.method }
  every(1.hour, 'Task 9') {Class.method}
  every(30.minutes, 'Task 10') {Class.method}
  every(30.minutes, 'Task 11') {Class.method}
  every(1.hour, 'Task 12') {Class.method}
end

I'm not quite sure where this is coming from. Maybe Sidekiq isn't removing the keys from the database once the job is complete?

Another potential helpful piece of information is that I'm running 4 workers/servers. Here is my Redis configuration:

if (Rails.env == "production" || Rails.env == "staging")
    redis_domain = ENV['REDIS_DOMAIN']  
    redis_port   = ENV['REDIS_PORT']

    redis_url = "redis://#{redis_domain}:#{redis_port}"

    Sidekiq.configure_server do |config|
      ActiveRecord::Base.establish_connection(
          adapter: "postgresql",
          encoding: "unicode",
          database: ENV["RDS_DB_NAME"],
          pool: 25,
          username: ENV["RDS_USERNAME"],
          password: ENV["RDS_PASSWORD"],
          host: ENV["RDS_HOST"],
          port: 5432
      )

      config.redis = {
        namespace: "sidekiq",
        url: redis_url
      }

    end

    Sidekiq.configure_client do |config|
      config.redis = {
        namespace: "sidekiq",
        url: redis_url
      }
    end
end

Anyone know why this could be happening?

Answer 1

Historical job metrics are stored per-day, for the past 5 years. You are seeing those 4-6 keys/day. This gives you the nice metrics on the Web UI's Dashboard.