We have an application which users can upload content to. We must run a job exactly N hours after a piece of content is uploaded.
I am wondering what is the best way to schedule jobs based off events such as these?
One option we can think of is we must persist an event in a database that records when the upload happened and run some check on this table every second to see if any events are exactly N hours old? This does not seem like a great solution.
Currently we are using node-Cron for other jobs in the application but it seems this does not support the use case we are looking for. We also came across agendaJS but have been unable to make it work with Typescript.
You can just use setTimeout() to set a timer for N hours after the event occurs. Nodejs is very efficient with zillions of timers so it's no big deal to just use a timer for each one of these.
For example to set a timer for 3 hours from now:
setTimeout(() => {
doSomething();
}, 1000 * 60 * 60 * 3);
To handle the case of your server restarting, you will have to have persisted in the database when each item should run its job and record when a job gets run so that upon server restart, you can query the items that still need their job to run and walk through each setting a timer for each one at its appointed time.
In case you're curious how nodejs handles zillions of timers efficiently, it puts all the timers into a sorted, linked list. The timer that should fire the soonest is at the head of the list.
As part of the process of running the event loop in nodejs, it checks to see if the time has been reached for the timer at the head of the list. If so, it calls the callback associated with that timer and removes it from the list.
When new timers are scheduled, they are inserted into the linked list in sorted order. In this way, there is no regular runtime penalty for millions of timers because all the event loop is doing is checking the head of the linked list to see if that timer's timer has been reached and that operation does not care how many timers are in the linked list.
There would be a slight extra cost when scheduling a timer if there are zillions of timers because the insertion of a new timer into the list in its proper sorted position will take slightly longer, but that is generally not an issue because it's a one time cost per timer. Note, any scheduling tool will have some sort of initial cost for setting up the timer.