I want to have discard in my fragment shader prevent that fragment from writing to the stencil buffer; I had an idea for a visual effect in my game, and it might be implementable if I can rely on that behaviour for discard.
My fragment shader doesn't do anything very fancy; it just samples a texture, and if the alpha is near 0, it discard otherwise it writes.
void main()
{
vec4 texColor = texture(tex, texCoord);
if(texColor.a < 0.5f)
{
discard;
}
FragColor = texColor;
}
However, it sounds like I cannot rely on discard preventing the stencil buffer from getting written to:
If Stencil Testing is active, then discarded fragments can still affect the stencil buffer. The stencil test can modify the stencil buffer, even on a stencil or depth test failure. And since the stencil test happens before the depth test, the depth test failures cannot prevent the stencil test from updating the stencil buffer.
I do need the stencil test active while I'm rendering. Can I not rely on discard preventing writes to the stencil buffer?
By the normal rules of OpenGL's order of operations, the stencil test happens after the execution of the fragment shader. As such, a fragment shader which executes a discard statement will prevent writes to framebuffer images. With one exception.
That exception being that the order of operations can be changed by explicit request of the FS. If no such request is given, then things must proceed as defined by the original order (which is why using discard turns off early fragment tests as an optimization).
The paragraph in question is talking about how part of the operation called the "stencil test" includes potentially updating to the stencil buffer. That is, the stencil test can both discard a fragment and change the stencil buffer's value based on how it was discarded. This is unlike the depth test, which if it fails, never updates the depth buffer's value.