Using a Continuous Integration Server as a watchdog

Question

I am currently implementing a system that checks the status of my servers, eventually applies some fixes and alert the admisitrator in case of errors (basically a watchdog).

I used to do it in a java container with scheduled tasks. On the other side I have my build server (Teamcity) which is really powerful but is used 1hr max per day. I am considering implementing my watchdog as build tasks.

Surprisingly I haven't found any nice article talking about that. Does anyone have some feedback doing that? Anything that I should think of before doing that? Any cons?

So far my pros/cons are :

Pros

• CI server is powerful but this power is used only 1hr a day
• Watchdog is smarter if it knows what's running on the build server and doesn't get trigger when a project is being redeployed
• CI Server have an integrated notification system
• CI Server keep tracks of the build log and have a nice interface to show what's right/wrong
• Tasks can be run manually as well as being scheduled

Cons

• Deployment is stuck can mean monitoring is stuck
• Harder to keep track of what happened in previous checks

Please enrich this list!

Thanks

Answer 1

You can achieve whatever you want to do using teamcity. However what you actually need is a monitoring service like nagios/icinga . We use both tools in our team and I have normally found that the

• nagios/icinga are much quicker to trigger
• can be run repeatedly at very quick intervals(2-3 sec in some cases)
• can be setup to run event handlers on failed tasks also. The only problem I see in teamcity is that you can setup build steps to run anytime/"previous step succeeded" but there is no hook to run a build step "only if the previous step failed"
• Also teamcity's revenue model is agent based. So you are constrained on the max number of agents you can run at any given time. There is no such constraint on monitoring systems. Infact they are free