Most efficient way to run a function on every time a timer finishes, but with a large number of timers (thousands/millions)

Question

I'm running a service where users upload a duration and a function must be repeatedly executed every time the timer runs out.

For example, the user says "Run every 5 minutes" and then this function must be run every 5 minutes. This is done through an API.

For a small numbers of timers, this is trivial:

func doEvery(d time.Duration, f func(time.Time)) {
    for x := range time.Tick(d) {
        f(x) // Run the function every d duration
    }
}

I can run each timer in a goroutine, and that works fine. I can start and stop everything with some basic WaitGroups and sync functionality.

But what if I have thousands or millions of timers? I can create a goroutine for each one, but that feels very inefficient. This isn't Erlang.

Should I have multiple work queues, sorted by the "delay", and simply allocate more workers for the more frequent functions? If the timer isn't ready, then it's put back on the queue.

This also isn't ideal, because the workers are busy-waiting (popping off, checking times, pushing on the queue) rather than blocking until the next timer finishes.

Maybe I could have some sort of map, indexed by the remaining duration? I'm not sure what the best approach is here.

Answer 1

I recently built this kind of solution where the user gets notification based on their interval settings. I make a worker pool model using rabbitMQ and two go script. Where one go script uses below cron module to create jobs in a message queue

https://github.com/robfig/cron

Another go script is the worker script consuming message queue and taking action. To scale, I'm running multiple instances of worker script. Autoscale of worker script can be done by the number of messages in rabbitMQ