How to mask multiple objects using stencil masking

Question

I am drawing two different stencil masks at the same location and would want my mask 1 to only affect shape1 and Mask 2 to affect shape 2.

This is the code.

glClearStencil(0);
glStencilMask(~0);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);

// Render all stencil masks

// First geometry Mask
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP);  // replace stencil buffer values to ref=1
glStencilFunc(GL_NEVER, 1, 0xff); // never pass stencil test
glStencilMask(0xFF);
DrawGeometry1();


// Second geometry Mask
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP);  // replace stencil buffer values to ref=1
glStencilFunc(GL_NEVER, 2, 0xff); // never pass stencil test
glStencilMask(0xFF);
DrawGeometry2();

//Draw first shape to be masked by first geometry Mask

glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);  
glStencilMask(0x00);
glStencilFunc(GL_EQUAL, 1, 0xff);
DrawShape1();


//Draw first shape to be masked by fsecond geometry Mask

glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);  
glStencilMask(0x00);
glStencilFunc(GL_EQUAL, 2, 0xff);
DrawShape1();

this is the workflow

1. draw both the masks.
2. draw shape 1
3. draw shape 2

the shape 2 is only affected by geometry mask 2 but the shape 1 which was drawn first gets affected by both the masks.

Answer 1

After rendering the masks, the stencil buffer has 4 possible values:

• 0 where noting is drawn
• 1 where mask 1 is drawn
• 2 where mask 2 is drawn
• 3 where both masks are drawn

When you draw the shapes, you just want take into account the masks of the shape. However, this means that you need to draw Shape 1 where the stencil buffer content is either 1 or 3, and Shape 2 where the stencil buffer content is either 2 or 3.
You only need to consider the 1st bit of the stencil buffer when drawing shape 1 and only the 2nd bit of the buffer when drawing shape 2.

Set the mask to either 1 or 2 when calling glStencilFunc:

//Draw first shape to be masked by first geometry Mask

glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);  
glStencilMask(0x00);
glStencilFunc(GL_EQUAL, 1, 1);   // <----
DrawShape1();


//Draw first shape to be masked by fsecond geometry Mask

glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);  
glStencilMask(0x00);
glStencilFunc(GL_EQUAL, 2, 2);   // <----
DrawShape1();