Search code examples
c++openglglm-mathstencil-buffer

How do values get drawn to the stencil buffer in this example?

I have a serious problem understanding how stencil buffering in OpenGL works. I followed the examples of a tutorial and the code works of course, but I really don't get at what points I'm writing to the stencil buffer. To me it seems as I'm setting up the stencil tests, masks and everything, but then I just call glDrawArrays(...) and write the triangles to the output without applying the stencil buffer at all. I also don't understand why I clear the buffer using glClear(GL_STENCIL_BUFFER_BIT) before I even draw the triangles. I would be very thankful if somebody could enlighten me.

// Draws the initial cube facing upwards
glDrawArrays(GL_TRIANGLES, 0, 36); 

//enable stencil testing
glEnable(GL_STENCIL_TEST);

// Draws the floor between the upward cube and downward cube 
glStencilFunc(GL_ALWAYS, 1, 0xFF); /*sets the test function, mask and ref value*/
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE); /*defines what happens on restults*/
glStencilMask(0xFF); /*Mask that is applied on the bits before they are written to the stencil buffer*/
glDepthMask(GL_FALSE);
glClear(GL_STENCIL_BUFFER_BIT); /*Why do I clear the buffer here?!?*/
glDrawArrays(GL_TRIANGLES, 36, 6);

// Draws the downward cube (reflection)
glStencilFunc(GL_EQUAL, 1, 0xFF);
glDepthMask(GL_TRUE);
glStencilMask(0x00);

model = glm::scale(
glm::translate(model, glm::vec3(0, 0, -1)),
glm::vec3(1, 1, -1)
);
glUniformMatrix4fv(uniModel, 1, GL_FALSE, glm::value_ptr(model));

glUniform3f(uniColor, 0.3f, 0.3f, 0.3f);
glDrawArrays(GL_TRIANGLES, 0, 36);
glUniform3f(uniColor, 1.0f, 1.0f, 1.0f);

glDisable(GL_STENCIL_TEST);
Solution

  • The third value passed to glStencilOp (GL_REPLACE) tells OpenGL to set the stencil buffer to the ref value previously specified in glStencilFunc whenever both the stencil test and the depth test are passed. In this case: 1.

    So then the call to glDrawArrays updates the stencil buffer to 1 everyplace where it draws successfully (i.e., passes both tests).

    (And the reason you clear it first, is to get rid of any 1's that happened to be there when you started: you don't want them acting as false positives where you didn't draw triangles).

    See https://www.khronos.org/registry/OpenGL-Refpages/gl2.1/xhtml/glStencilOp.xml for details.