I am looking at various buffer/heap/stack protection technologies such as PAX, DEP, NX, CANARIES, etc
And a new one SMEP - http://vulnfactory.org/blog/2011/06/05/smep-what-is-it-and-how-to-beat-it-on-linux/
Assuming that i am using the latest kernel on the latest mainstream processor Assuming i recompile all my apps with various compiler protections Assuming i run a good DEP, ASLR NX bit whatever protection
Is it reasonable to say that most buffer-overflow attack will fail? And this has been solved for future systems?
As corollary, is it true to say that as currently implemented, current Win7 systems are hopelessly vulnerable to ROT and other techniques particularly on 32bit apps ?
p.s. I am purposely not going into the "managed code is safe" argument here
[disclaimer] I am not advocating sloppy coding. I realize there are many other attacks. I know that existing systems far outnumber the "idealized" security configuration
No, buffer-overflow is not a solved problem, at least in unmanaged code.
The basic problem with all the technologies you listed (DEP/NX, ASLR, Canaries, SMEP) is that they happen after the fact: they cannot solve the buffer overflow problem, because they do not prevent the root cause - buffer overflow and memory corruption has already happened.
All of the schemes are simply 'good' but unreliable heuristic ways to attempt to either detect the fault after corruption happens but before it becomes a working exploit, or just make it too hard to engineer a reliable working exploit.
e.g. DEP, ASLR, and canaries on Windows Vista and 7 can be circumvented by ROT plus other techniques, and therefore just raise the difficulty bar of creating a working exploit. But, the difficulty increase is quite significant. This is the point of the techniques.
If you are interested in solving the problem at its root by preventing the original buffer corruption, I think that would be most productive, but it would probably bring us back to the discussion of managed code...