How can I switch redis host/pool in sidekiq?

Question

I have multi-tenant rails application with apartment gem I can successfully switch database tenant in each worker using apartment-sidekiq. But sidekiq workers are using same redis server for all the tenant. I want to have complete isolation on redis as well. Is there a way in which I can use the separate redis host for each tenant without impact much code changes using sidekiq middeware.

Sample Tenant config

 TENANT_POOL = {
  tenant_1: ConnectionPool.new { Redis.new(url: ENV['REDIS_URL1']) },
  tenant_2: ConnectionPool.new { Redis.new(url: ENV['REDIS_URL2']) }  
 }

Eg. Worker

class SendMailWorker
  include Sidekiq::Worker
  def perform(args)
    puts "In Worker Tenant: #{Apartment::Tenant.current}, #{Sidekiq.redis_pool.inspect}"
  end
end     

Client middleware

class MyMiddleware::Client::ChangeRedisPool
  def call(worker_class, job, queue, redis_pool)
    #Can I change redis pool/client here to execute job in separate redis 
    yield
  end
end

Server middleware

class MyMiddleware::Server::ChangeServerRedisPool
  def call(worker, job, queue)
    #Can I change redis pool/client here to execute job in separate redis 
    yield
  end
end

I know we can execute worker with specific connection pool like following, but this will end up changing code at every sidekiq worker invocation.

 Sidekiq::Client.via(TENANT_POOL[:tenant_1]) { SendMailWorker.perform_async("d") } # run on tenant 1 sidekiq
 Sidekiq::Client.via(TENANT_POOL[:tenant_2]) { SendMailWorker.perform_async("d") } # run on tenant 1 sidekiq

How we can achieve multi tenancy with sidekiq and separate redis?

Answer 1

You need to start a separate Sidekiq process for each Redis. Your client can dynamically target a Redis pool like this:

SendMailWorker.set("pool" => POOLx).perform_async(...)

Look at the code for Sidekiq::Client and Sidekiq::Worker for details.