OperGL stencil buffer not acting how I expect

Question

I have a simple UI widget system and I'm using the stencil buffer to act as a clipper so that children of a widget can't be drawn outside the widget. Basically, the stencil value for everything is inside the bounds for this widget is incremented. Then, anything drawn after I make the clipper must be within the box.

The clipper constructor looks like this:

// Stencil buffer and ClipperStack.size() start at 0
// Increment any pixel in the rect on my current recursion level 
glStencilFunc(GL_EQUAL, ClipperStack.size(), 0xFF);
glStencilOp(GL_KEEP, GL_INCR, GL_INCR);
ClipperStack.push_back(Rect);
// only draw to stencil buffer
glColorMask(0, 0, 0, 0);
glStencilMask(1);
glBegin(GL_QUADS);
    glVertex2f(Rect.Left, Rect.Top);
    glVertex2f(Rect.Left, Rect.Bottom);
    glVertex2f(Rect.Right, Rect.Bottom);
    glVertex2f(Rect.Right, Rect.Top);
glEnd();
// Stencil clipper drawn, 
glColorMask(1, 1, 1, 1);
glStencilMask(0);
// now only draw stuff that's that has the right clipper value
glStencilFunc(GL_EQUAL, ClipperStack.size(), 0xFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);

When the clipper goes out of scope the destructor runs, which looks like this:

// Decrement anything we previously incremented
glStencilFunc(GL_EQUAL, ClipperStack.size(), 0xFF);
glStencilOp(GL_KEEP, GL_DECR, GL_DECR);
// Get the old rect
sf::FloatRect Rect = clipperStack.back();
ClipperStack.pop_back();
// Only draw to stencil buffer
glColorMask(0, 0, 0, 0);
glStencilMask(1);
glBegin(GL_QUADS);
    glVertex2f(Rect.Left, Rect.Top);
    glVertex2f(Rect.Left, Rect.Bottom);
    glVertex2f(Rect.Right, Rect.Bottom);
    glVertex2f(Rect.Right, Rect.Top);
glEnd();
// now draw on regular color buffer again, 
// stencil buffer should be the same as before constructor call
glColorMask(1, 1, 1, 1);
glStencilMask(0);
glStencilFunc(GL_EQUAL, ClipperStack.size(), 0xFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);

However, when I run this only the direct children of my root widget are drawn. The children of children aren't drawn at all. I've tried a bunch of variations of this and I keep doing something wrong. I don't know where I'm going wrong with this.

Answer 1

In both the constructor and destructor I think you need to set the glStencilMask() to set every bit of the stencil buffer. E.g. if you have an 8-bit stencil buffer you want to use glStencilMask(0xFF);

Of course, if you've only got a 1-bit stencil buffer your code won't work at all, since you're trying to increment the stencil value for each level of sub-widget.