In a pixel shader you can discard a pixel but I would imagine even a fast-fail shader called for every pixel takes a non-trivial time? Is there any way a vertex shader can discard an entire triangle... I am fairly sure a VS can't access the primitive but are there any tricks by which we can get the same result?
Talking SM 3.0 here - for completeness discussion on newer versions is welcome.
The geometry shader can do that trivially as you probably know, but no such thing in SM3.
It's harsh for the vertex shader to identify a triangle as such, since all the vertex shader sees is the individual vertices. But... in principle it's possible.
If you set the w coordinate to zero, a point will be projected to infinity. Set the w coordinates of all points to zero, and the entire triangle is at infinity, so it won't be rendered.
Alternatively, you could probably set gl_ClipDistance to zero or a value below zero (never really used those, but I guess this should work). That would mark the vertex as "behind" a clipping plane. If all three vertices are behind a clipping plane, the triangle is not visible.
You would need a way to identify the vertices that belong to the triangle that you want to discard, however... and that will not be easy unless you have a very special situation (such as for example knowing that you want to discard all triangles in some given bounding box, or all triangles in which the vertex attribute #5 is zero, or whatever).