So I'm kind of stuck in a pickle here, and not sure what route to take. My application as of right now receives POST requests from our machines that whenever one goes down, we call them alerts. Some of the issues that get posted to my server have specific steps that the IT guys have to follow in order to resolve the machines issue. Some of those steps are to wait for a specific amount of time, such as 15, 30, or 60 minutes. After the time frame has passed the alert is moved onto the next step, and so on until the machine is back up and running. As of right now we are doing a setTimout() for each alert that comes through with a wait time, but the problem is that it is happening on the client side which is causing a lot of overhead due to some of these timeouts being duplicated.
Obviously the way that it is being set up right now is not optimal and not the most appropriate solution, so I was thinking about doing this check from the server side instead. My first thought was to just set the timeouts on the server side, but some issues with this approach is that if the server ever had down time or crashed, these timeouts would get erased. My other thought was to just do a cron job every minute from the server to check if any of the alerts have passed their wait time and then update the alerts from there. This would work in the instances where the server has down time or crashed and needs a reboot, but I'm worried that this would be overkill for the server.
I'm really just looking for some good solutions here as I feel like I'm stuck in deciding on which route I should take. I'm leaning towards the cron job option but am worried about it overloading the server.
I don't think creating an interval for the cron job for longer than 1 minute would work because if two or more alerts came in at different times they would need to update after exactly 15, 30, or 60 minutes from the time they were initiated.
If anyone has any suggestions on ways that I could accomplish this, that would be very helpful. Thanks!
Based on what the script you describe does, a cron job running it every minute is not an overkill for the server. However, I would advise you to make use of a process control system likesupervisord. You can spawn workers with it and they'll restart automatically after they finish executing. This way you'll have even smaller intervals between each check and higher reliability as you have multiple workers checking.