I'm trying to use stencil buffers on ID3D11Texture2D to overlay a portion of one texture on another texture. I plan to do this by drawing the desired shape on the stencil buffer and then copying the texture. Please suggest some good tutorial with sample code for 2D Textures for learning the usage of stencil buffer. (or some alternate way of doing this)
Stencil mask 101:
- All mainstream GPUs use an 8 bit stencil buffer, and they are tied to the depth buffer, so you must choose a buffer format that includes stencil (D24S8 primarily)
- Clear depth and stencil both for the beginning of each frame to avoid inter-frame performance problems *(see note below)
- Unless you know what you are doing, use 0xFF for the read and write mask at all times.
- For performance reasons, not have depth-writing enabled when doing stencil-read tests
- For performance reasons, if at all possible, keep the stencil read tests restricted to ==0 and !=0, and clear to zero
- If you are using D3D9, you can use scissor rects combined with clearing the stencil buffer, this is quite a bit faster than rendering a quad manually to set the buffer back to zero.
- There is no way to access stencil information in a shader, so postprocess effects can't determine the stencil state of adjacent pixels without the mask also being in a normal rendertarget and sampled as texture.
- Stencil buffer values can be modified from the following conditions:
- The stencil test failed
- The depth test failed
- The stencil and depth tests both passed
- Whichever condition is first determines the action, which can be
- keep (leave the stencil value alone)
- replace (replace the stencil value with the 'stencilref' value)
- increment (add 1 to the current buffer)
- decrement (sub 1 from the stencil buffer)
- zero (set the value to directly zero, useful when one of the other conditions needs to use the current stencilref value)
*Failure to follow most of these rules usually results in worst case stencil performance: Your pixel shader executing for every pixel, and not performing an early reject for the stencil masked pixels. In addition on older Geforce cards this can also result in the breaking of the early depth reject for all subsequent draw calls, which won't be fixed until the next frame or depth buffer & stencil is cleared.
Stencil masks are actually pretty easy to generate:
- Clear the stencil buffer
- Enable the stencil state for stencil writing ('replace' with stencilref 1 or 'increment' instead of 'keep')
Draw a polygon. Note that using 'discard' in the shader will prevent the stencil from being updated, which is how you load a texture into the stencil buffer (but only as a binary mask)
Disable stencil writes, enable stencil test
- (Ideally) also disable depth writes
- Draw your effect that requires stencil masking