Search code examples
opengl-esblending

OpenGL transparency in texture when render with stencil buffer

The question has been updated thanks to the comments.

Screenshot of how textures overlap
Screenshot of how textures overlap

To draw 2 points with brush texture using the stencil buffer to avoid textures transparency overlap, the following code is used:

    glEnable(GL_STENCIL_TEST.gluint)

    glClear(GL_STENCIL_BUFFER_BIT.gluint | GL_DEPTH_BUFFER_BIT.gluint)

    glStencilOp(GL_KEEP.gluint, GL_KEEP.gluint, GL_REPLACE.gluint)

    glStencilFunc(GL_ALWAYS.gluint, 1, 1)
    glStencilMask(1)

    glDrawArrays(GL_POINTS.gluint, 0, 1)

    glStencilFunc(GL_NOTEQUAL.gluint, 1, 1)
    glStencilMask(1)

    glDrawArrays(GL_POINTS.gluint, 1, 1)

    glDisable(GL_STENCIL_TEST.gluint)

And stencil buffer works, however, each point fill a full rectangle in the stencil buffer, but a texture image has transparency. So maybe texture used in the wrong way?

The texture is loaded like this

    glGenTextures(1, &gl_id)
    glBindTexture(GL_TEXTURE_2D.gluint, gl_id)
    glTexParameteri(GL_TEXTURE_2D.gluint, GL_TEXTURE_MIN_FILTER.gluint, GL_LINEAR)
    glTexImage2D(GL_TEXTURE_2D.gluint, 0, GL_RGBA, gl_width.int32, gl_height.int32, 0, GL_RGBA.gluint, GL_UNSIGNED_BYTE.gluint, gl_data)

Blending set as

    glEnable(GL_BLEND.gluint)
    glBlendFunc(GL_ONE.gluint, GL_ONE_MINUS_SRC_ALPHA.gluint)

Could you please advice where to look in order to fill 1s in stencil buffer by exactly not transparent area of brush image?

Solution

  • I recommend to discard the transparent parts of the texture in the fragment shader. A fragment can be completely skipped in the fragment shader by the discard keyword.
    See Fragment Shader - Special operations.

    Use a small threshold and discard a fragment, if the alpha channel of the texture color is below the threshold:

    vec4  texture_color = .....;
float threshold = 0.01;
if ( texture_color.a < threshold )
    discard;

    An other possibility would be to use an Alpha test. This would be only available in OpenGL compatibility profile, but not in core profile or OpenGL ES.

    See Khronos OpenGL-Refpages glAlphaFunc:

    The alpha test discards fragments depending on the outcome of a comparison between an incoming fragment's alpha value and a constant reference value.

    With the following alpha test, the fragments whos alpha channel is below the threshold are discarded:

    float threshold = 0.01;
glAlphaFunc(GL_GEQUAL, threshold);
glEnable(GL_ALPHA_TEST)