How to render a 2D UI on top of a 3D scene before rendering the 3D scene?

I have a 2D texture that contains 2D overlay. The texture itself is mostly blank (transparent) with a few parts containing some data.

What I currently do is render the whole 3D scene, disable the depth buffer and render the 2D quad on top of it:

 // render 3D scene
 context->OMSetDepthStencilState(_noDepthTestState.Get(), 1); // disable depth test
 // render 2D quad on top of the whole viewport
 context->OMSetDepthStencilState(nullptr, 1); // restore default

The _noDepthTestState variable is a ID3D11DepthStencilState created with the following descriptor:

   D3D11_DEPTH_STENCIL_DESC desc{};
   desc.DepthEnable = false;
   desc.DepthWriteMask = D3D11_DEPTH_WRITE_MASK_ALL;
   desc.DepthFunc = D3D11_COMPARISON_LESS;
   desc.StencilEnable = true;
   desc.StencilReadMask = 0xFF; 
   desc.StencilWriteMask = 0xFF;
   desc.FrontFace.StencilFailOp = D3D11_STENCIL_OP_KEEP;
   desc.FrontFace.StencilDepthFailOp = D3D11_STENCIL_OP_INCR;
   desc.FrontFace.StencilPassOp = D3D11_STENCIL_OP_KEEP;
   desc.FrontFace.StencilFunc = D3D11_COMPARISON_ALWAYS;
   desc.BackFace.StencilFailOp = D3D11_STENCIL_OP_KEEP;
   desc.BackFace.StencilDepthFailOp = D3D11_STENCIL_OP_DECR;
   desc.BackFace.StencilPassOp = D3D11_STENCIL_OP_KEEP;
   desc.BackFace.StencilFunc = D3D11_COMPARISON_ALWAYS;

I would like to render the 2D overlay quad before rendering the 3D scene, and that 3D objects will be drawn only in places where the 2D overlay is blank.

Is there an efficient way to achieve this behavior? what is the correct configuration of depth / stencil states?

Solution

It depends on the way your main 3D scene is rendered.

Assuming you don’t use stencil buffer, and you using D3D11_COMPARISON_LESS depth comparison, draw the quad in the following way.

Your vertex shader should output quad vertices [ ±1, ±1, 0, 1 ]. This makes GUI to be the closest to the camera, early Z rejection should clip occluded pixels in the 3D scene before PS stage, saving some GPU resources (I’m assuming that’s why you want to render the 2D first).

Your pixel shader should read from your GUI texture, compare alpha to some threshold, and if it’s small enough, call discard. Discarded pixels don’t change any buffers, neither color nor depth/stencil.

This will work OK if your GUI is made of axis-aligned rectangles, or if you don’t use MSAA. However, if you do have curved/diagonal edges of the GUI, and using MSAA, you won’t be happy with the result. Fixing that is possible but way more complicated. You gonna need to somehow produce per-pixel SV_Coverage values when rendering your GUI texture.