Sending email using sidekiq, redis

Question

I have a question about sidekiq in ruby. I'm building a system to send campaign email, 10k - 50k emails/campaign. For now i'm using Redis and Sidekiq, for every email I create a worker, like this:

receivers.each do |receiver|
    begin
      email_sending_log = EmailSendingLog.create(email: 
        receiver["email"], sender: sender["email"], content: 
        content["value"], status: "REQUESTED")
        EmailWorker.perform_async(adapter, receiver, sender, content, subject, email_sending_log.id.to_s)
    rescue Exception => e
    end
end

But it's so slow, Do you guys have any idea to speed up my system? Or some example to speed up sidekiq, redis ?

Answer 1

Sending campaign emails is pretty standard task - one has to send N email(your N=10k..50k).
Beginners choose two opposite ways: one background job with N-loop OR N background jobs with one sending.

Both are bad:

• Long N-loop is vulnerable to errors and force shutdowns(app deploy etc) - a lot of messages might be send twice and a lot of ones might be lost.
• N-jobs takes a lot of time due to job processing overhead(your case).

What to do?

Batches. Use M jobs with K-loop each where M*K=N. Example: 100+ jobs with 100-loop each for your case. Try different M, K and choose best one for you. It's not a silver bullet but a golden mean of the bad opposite ways.

Email sending services. If your campaign is marketing emails(I think it is), the best way is to use email sending services like Sendgrid, SES, Mailchimp, Mailgun, Elasticemail and so on.
They allow you to create email template(with substitutions) per campaign and then do whole campaign sending with just one API call(with addresses and substitutions as parameters of the call). It's fastest and the most reliable way, moreover email sending services provide additional benefits. The only drawback is it is paid.