Search code examples
c++openglopengl-3stencil-buffer

glStencilFunc(): why is GL_ALWAYS != GL_EQUAL union GL_NOTEQUAL?


I want to be able to clip my postprocessing image passes to specific regions so that effects such as a blur would only affect theses regions

In order to do that i use the stencil buffer and my pipeline is as follows :

  1. Render some objects to the stencil buffer only, writing 1s
  2. Render some objects where the stencil value equals 1 (this works)
  3. Render some objects whatever there is in the stencil buffer
  4. Run postprocess passes (by drawing a quad with the image resulting of the 3 previous step as a bound texture) where the stencil value equals 1 (or always, depending on an attribute of my effects)

The results i get :

  • Black image when postprocess involves stencil buffer
  • 'Good' image when it does not
  • An image with non null values only outside the masks when i change `glStencilFunc(GL_EQUAL, 1, 0xFF);` to `glStencilFunc(GL_NOTEQUAL, 1, 0xFF);`

What strikes me is the fact that the image obtained with glStencilFunc(GL_ALWAYS, 1, 0xFF); is not even equal to the union of the other two : the one with glStencilFunc(GL_EQUAL, 1, 0xFF); is all black.

What is wrong with this code ?

gl->glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, objectsTex, 0);
gl->glClear(GL_COLOR_BUFFER_BIT);

// =================    Masks    ===================
gl->glEnable(GL_STENCIL_TEST);
gl->glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE); // Disable color buffer writing
gl->glStencilFunc(GL_ALWAYS, 1, MASKSBITPLANE);
gl->glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
gl->glStencilMask(MASKSBITPLANE); // Write values as is in the stencil buffer
gl->glClear(GL_STENCIL_BUFFER_BIT);
for (const auto& scobjptr : renderGroup->getRenderGroupObjects().getMaskObjects()){
     renderBlankSceneObject(scobjptr, gl, glext);
}

// =================    Masked   ===================
gl->glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE); // Enable color buffer writing
gl->glStencilFunc(GL_EQUAL, 1, MASKSBITPLANE);
gl->glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
gl->glStencilMask(0x00); // Disable writing to the stencil buffer
for (const auto& scobjptr : renderGroup->getRenderGroupObjects().getMaskedObjects()){
     renderSceneObject(scobjptr, gl, glext);
}

// ================= Raw objects ===================
gl->glDisable(GL_STENCIL_TEST);
for (const auto& scobjptr : renderGroup->getRenderGroupObjects().getRawObjects()){
     renderSceneObject(scobjptr, gl, glext);
}

// ================= Postprocess ===================
auto& shaderEffects(renderGroup->shaderEffects());
if (renderGroup->areShaderEffectsMasked()){
     gl->glEnable(GL_STENCIL_TEST);
     gl->glStencilFunc(GL_EQUAL, 1, MASKSBITPLANE);
     gl->glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
     gl->glStencilMask(0x00); // Disable writing to the stencil buffer
}
for (auto it(shaderEffects.begin()); it != shaderEffects.end(); ++it)
{
     gl->glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, currentImageTex, 0);
     gl->glClear(GL_COLOR_BUFFER_BIT);

     // postprocess
     gl->glUseProgram(shaderEffect->program().programId());
     gl->glUniform1f(shaderEffect->m_timeLocation, m_time.elapsed());
     gl->glActiveTexture(GL_TEXTURE0 + GLShaderEffect::PROCESSED_IMAGE_TEXTURE);
     gl->glBindTexture(GL_TEXTURE_2D, processedTexture);
     // some glUniform* calls
     updateUniforms(gl, shaderEffect->ressourceClientsCollection());
     // some glActiveTexture + glBindTexture calls
     bindTextures(gl, shaderEffect->ressourceClientsCollection());
     glext->glBindVertexArray(shaderEffect->vao());
     gl->glDrawElements(GL_TRIANGLES, shaderEffect->elementsCount(), GL_UNSIGNED_INT, nullptr);

     swap(currentImageTex, objectsTex);
}

Solution

  • The answer : I didn't restore the context after drawing, e.g. disable stencil testing at the end of my render pass. I use Qt to blit my framebuffer into a widget and the stencil test was still active with another stencil buffer attached : that is why the screen got black.

    Conclusion : Always restore the context to its previous state when you use a framework