In RTOses like vxworks whenever we create a task the stacksize is specified . Can we write a routine in C which checks if the stack is overflowing or not for the task ?
Look at your compiler they often let you add prelude functions to do this or they they might even check it themself unless you manipulate the stack pointer register.
And check if the operating system allows you to install "guard-pages". Mark the last page in your threads stack as non-read/non-write and catch the SIGSEGV signal and use a OS/CPU specific way to find out if it is the guard-page that failed. For this to work you must be sure that the stackframe of a function (stack passed parameters, local variables and alloca allocated space) is always less then a page size otherwise you can skip over the "guard-page" This is the best way to handle it as it has no runtime overhead during normal processing.
You see this highly OS/CPU/Compiler dependent. But i'm pretty sure that google will find useable code and helpers for this technique for all systems as it is a pretty common technique for low level programmers (for example runtime or interpreter implementors).