Check a stencil bit, and write to another bit

I'm rendering reflective surfaces in my 3D engine. I need to perform two stencil operations. First, I draw out the reflective surface with depth testing to find visible areas of the surface. I then need to draw out models into a G-Buffer, which is also stencilled to find areas to draw my skybox.

Draw Surface: always draw, write bit #1
Draw Models: draw only if bit #1 is set, write bit #2

How would I do this using OpenGL? I'm unsure of the relationship between the glStencilFunc ref and mask values, and the glDepthMask value.

Solution

The docs are quite specific, but it's not always intuitive or obvious what to do if you simply want to create and then activate a mask. I use these as a simple starting point...

Create the mask

Clear the stencil buffer to zeroes and write 1s to all pixels you draw to.

void createStencilMask()
{
    // Clear the stencil buffer with zeroes.
    // This assumes glClearStencil() is unchanged.
    glClear(GL_STENCIL_BUFFER_BIT);

    // Enable stencil raster ops
    // 1. Enables writing to the stencil buffer (glStencilOp)
    // 2. If stencil test (glStencilFunc) fails, no frags are written
    glEnable(GL_STENCIL_TEST);

    // sfail GL_KEEP - keep original value if stencil test fails
    // dpfail GL_KEEP - also keep if the stencil passes but depth test fails
    // dppass GL_REPLACE - write only if both pass and you'd normally see it
    //                     this writes the value of 'ref' to the buffer
    glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);

    // func GL_ALWAYS - the stencil test always passes
    // ref 1 - replace with ones instead of zeroes
    // mask 1 - only operate on the first bit (don't need any more)
    // Assumes glStencilMask() is unchanged
    glStencilFunc(GL_ALWAYS, 1, 1);
}

Call the above function, and draw your stuff. You can set glDepthMask and glColorMask so this doesn't actually affect the current colour/depth and your scene, only the stencil buffer.

Draw, using the mask

Only draw to pixels with 1s from the previous step.

void useStencilMask()
{
    // Enable stencil raster ops
    glEnable(GL_STENCIL_TEST);

    // Just using the test, don't need to replace anything.
    glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);

    // Only render if the current pixel stencil value is one.
    glStencilFunc(GL_EQUAL, 1, 1);
}

Draw and then leave the test disabled glDisable(GL_STENCIL_TEST) when you're done.

Reading and writing different bits

Now to focus on your question...

This bit's a little tricky because the same ref value of glStencilFunc() is used in both the stencil test and as the value to replace. However, you can get around this with masks:

Use mask in glStencilFunc() to ignore bits when testing/reading.
Use glStencilMask() to stop certain bits from getting written.

In your case you don't need to mask out the first bit from being written because it's already set. Simply update useStencilMask() to glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE) and glStencilFunc(GL_EQUAL, 3, 1);. Because mask is 1, only the first bit is used in the test for equality. However the whole ref of 3 (which is 0b11) is written. You could use glStencilMask(2) (which is 0b10) to stop the first bit from being written but it's already one so it doesn't matter.

You could also make use of GL_INCR which would set the second bit and remove the first. Or perhaps clear with ones and use GL_ZERO to mark both your bits.