Background jobs occur very frequently and eat memory

Question

I'd like to optimize my notification system, so here is how it works now:

Every time some change occurred on application, we're calling background job (Sidekiq) in order to compute some values and then to notify users via email.

This approach worked very well for a while, but suddenly we got memory leak as there were a lot of actions very frequently and we had about 30-50 workers per second so I need to refactor this.

What I would like to do is, instead of running worker immediately, to store it in array and perform bit later.

But I'm afraid that also will cause a problem, but just "delayed" problem.

I'm looking forward to hear more approaches and solutions as well.

Thanks in advance

Answer 1

So I found one very interesting solution:

I'm storing values to Redis directly as key - value, where the value is dataset with data I'd need later for computation. Then I'm using simple cron job, which occurs service which is responsible for reading data from Redis and computing them. I optimized Sidekiq workers to work only when cron is executed, everything works perfectly fine and even much faster then before.

I'm still eager to hear if there is any other approach/solution.

Thanks