Depth test not working correctly with stencil test

Question

I need to cut holes in the 3D surface using stencil buffer. Currently, everything works as expected, but the main problem is that holes are also visible through hills. How to prevent this behavior and hide holes if they are behind hills?

Current code:

        GLES20.glColorMask(false,false,false,false);
        GLES20.glDepthMask(false);
        GLES20.glEnable(GLES20.GL_STENCIL_TEST);
        GLES20.glStencilFunc(GLES20.GL_ALWAYS, 1, 0xFF);
        GLES20.glStencilOp(GLES20.GL_KEEP, GLES20.GL_KEEP, GLES20.GL_REPLACE);
        GLES20.glStencilMask(0xFF);
        GLES20.glClear(GLES20.GL_STENCIL_BUFFER_BIT);
        //drawing holes
        GLES20.glStencilFunc(GLES20.GL_EQUAL, 0, 0xFF);
        GLES20.glStencilMask(0x00);
        GLES20.glColorMask(true,true,true,true);
        GLES20.glDepthMask(true);
        //draw surface with elevation

Answer 1

A solution which provides the effect, can be achieved by Face Culling by a separated stencil test for front and back faces and (see glStencilFuncSeparate and glStencilOpSeparate). Sadly the back faces of the geometry have to be drawn in a separated step.

The process can be described in the following steps:

• Enable the depth test

• Disable the color buffer and enable the stencil test for setting the stencil mask

• Draw the "holes". This causes that the stencil buffer is set to 1 at the positions of the holes.

• Setup the stencil test for clearing the the stencil mask by backfaces

• Enable face culling for front faces

• Draw the geometry. This causes that the stencil buffer is cleared at positions which are covered.

• Enable face culling for back faces

• Enable the color buffer

• Draw the geometry

To make tha algorithm work, you have to draw all your primitives int the same winding order. And you have to tell OpenGL the direction by glFrontFace. Either clockwise GL_CW or counterclockwise GL_CCW.

GLES20.glStencilMask(0xFF);
GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT | GLES20.GL_DEPTH_BUFFER_BIT | GLES20.GL_STENCIL_BUFFER_BIT);

GLES20.glFrontFace(GLES20.GL_CCW);  // depends on your geometry "GL_CCW" or "GL_CW"
GLES20.glDisable(GLES20.GL_CULL_FACE);

GLES20.glEnable(GLES20.GL_DEPTH_TEST);
GLES20.glDepthFunc(GLES20.GL_LESS); // default

GLES20.glColorMask(false,false,false,false);

GLES20.glEnable(GLES20.GL_STENCIL_TEST);

GLES20.glStencilFuncSeparate(GLES20.GL_FRONT, GLES20.GL_ALWAYS, 1, 0xFF);
GLES20.glStencilFuncSeparate(GLES20.GL_BACK, GLES20.GL_ALWAYS, 1, 0xFF);
GLES20.glStencilOpSeparate(GLES20.GL_FRONT, GLES20.GL_KEEP, GLES20.GL_KEEP, GLES20.GL_REPLACE);
GLES20.glStencilOpSeparate(GLES20.GL_BACK, GLES20.GL_KEEP, GLES20.GL_KEEP, GLES20.GL_KEEP);

// draw the holes
// ....

GLES20.glStencilFuncSeparate(GLES20.GL_FRONT, GLES20.GL_EQUAL, 0, 0xFF);
GLES20.glStencilOpSeparate(GLES20.GL_FRONT, GLES20.GL_KEEP, GLES20.GL_KEEP, GLES20.GL_KEEP);
GLES20.glStencilFuncSeparate(GLES20.GL_BACK, GLES20.GL_ALWAYS, 0, 0xFF);
GLES20.glStencilOpSeparate(GLES20.GL_BACK, GLES20.GL_KEEP, GLES20.GL_KEEP, GL_REPLACE);

GLES20.glEnable(GLES20.GL_CULL_FACE);
GLES20.glCullFace(GLES20.GL_FRONT);

// draw the geometry the 1. time ("draw" back faces)
// ....

GLES20.glCullFace(GLES20.GL_BACK);
GLES20.glColorMask(true,true,true,true);

// draw the geometry the 2. time (draw front faces)
// ....