I currently have a list of servers/websites I need to monitor, once a minute.
Is there a good way to round robin through them every minute, with the ability to add/remove servers/websites at any point?
I have looked into RabbitMQ, but it does not seem feasible to consistently add in and consume a task every minute. The tasks are stored in MongoDB, perhaps another solution I thought of is to query through MongoDB, but that does not seem very efficient.
I don't think a message broker would be necessary. What I would do is actually get the tasks from the database and store them in memory at the beginning of the program. Create a task class and for each task in the database, replicate it in the task class and store all those task objects in a task array.
Have that list of servers/websites stored in a database.
Have your program query for the list of servers/websites from that database every minute to obtain the latest list of servers/websites. You can always go into your database and remove or add servers/websites.
Assuming you want to do all the tasks for each server/website, you basically execute all the tasks in the task array for each. It's essentially a for loop inside another for loop.
If that assumption is incorrect, you get the gist of what I mean.
Hope this helps.