When running one of our software, a tester was faced with the data execution prevention dialog of Windows.
We try to reproduce this situation on a developer computer for debugging purposes : with no success.
Does anyone know how to find what may cause the DEP protection to kill the application? Is there any existing tools available for this?
The DEP dialog will typically only show when you try to execute code from a region that you're not marking as executable. This might be caused by 'thunks' in a library you're using, e.g. ATL windowing. This problem is fixed in ATL 8.0.
A stack-trashing bug - for example, a buffer overrun - can also cause this problem, by setting the return address to a location that isn't executable. This might not cause an access violation but instead weird behaviour, if DEP is turned off for the process or not available on the hardware.
It might also happen if you throw a C++ exception or raise an SEH exception, and your structured exception handlers have been trashed by a buffer overrun.