We have a remote device that is controlled by a C program. This program often fails so we want to introduce some monitoring. Either the program crashes and exits or some bug happens and the program is not responding.
We want to set up a cron task or some other scripts for monitoring. What is the best modern practice for communication between C program and bash script or cron? We have access to the source code of this C program, so we can introduce some sort of error messages / events. Should they be written to a file that is later parsed by cron or some other monitoring script - or is this approach obsolete and there is some API or Linux messaging system for communication is such cases?
As a simple variant, use a watchdog. The program on regular intervals writes to a well-specified file. If the file haven't been updated in a long enough time, the program is unresponsive, and you should kill and restart it.
With that said, the best solution is of course to fix the problems that cause the crashes or unresponsiveness. You should have enough logging to be able to pinpoint exactly when and where things go wrong. And perhaps make sure there's a coredump available for crashes, so you can debug it.