I have a collection of executables that regularly update a collection of files every couple of minutes 24/7. I am thinking about writing a single monitoring program that will continuously check the last write time (using the function stat()) of all these files so that if any have not been updated recently enough it can ring an alarm. My concern though is that perhaps the very act of calling stat() may cause a program that is attempting to write to that file, to fail. Need I worry?... and if so is there an alternative way to achieve my goal?
Yes, a stat call can be thought of as atomic, in that all the information it returns is guaranteed to be consistent. If you call stat at the same instant some other process is writing to the file, there should be no possibility that, say, the other process's write is reflected in st_mtime but not st_size. [Addendum: per a comment below, my "should be no possibility" claim here isn't necessarily true.]
In any case, there's certainly no possibility that calling stat at the same instant some other process is writing to the file could cause that other process to fail. (That would be a serious and quite unacceptable bug in the operating system -- one of an OS'es main jobs is to ensure that unrelated processes can't accidentally interact with each other in such ways. This lack-of-interference property isn't usually what we mean by "atomic", though.)
With that said, though, the usual way to monitor a process is via its process ID. And there are probably plenty of prewritten packages out there to help you manage one or more processes that are supposed to run continuously, giving you clean start/stop and monitoring capabilities. (See s6 as an example. I know nothing about this package and am not recommending it; it's just the first one I came across in a web search.)
Another possibility, if you have any kind of IPC mechanism set up between your processes, is to set up a periodic heartbeat that each one publishes, so that a watchdog timer somewhere can detect a process dying.
If you want to keep monitoring your processes by the timeliness of the files they write, though, that sounds like a perfectly fine technique also.