Why GL_RASTERIZER_DISCARD does not enable me to write to stencil buffer?

Question

I want to draw outline with stencil buffer in 3 passes.

1. normal drawing pass
2. outline mask generating pass
3. outline drawing pass

the second pass does not need fragment shader to work, because it only writes to stencil buffer, so I use glEnable(GL_RASTERIZER_DISCARD); to prevent touching fragment shader at all.

Problem is that the mask is not being generated (that is my thought, maybe it is something else), thus outline quad fragments does not get discarded. It is drawn but on top of the quad that I want to be outlined. If that makes sense.

int main()
{
    //create window, gl context etc. etc...

    Shader colorShader("colorShader.glsl");
    Shader maskShader("maskShader.glsl");

    //Creates geometry for quad, also creates vertex array, vertex buffer and adds data glBufferData.
    //this code is tested and works, so it is not neccessery to show what it does internaly.
    Quad sceneQuad(colorShader);
    Quad maskQuad(maskShader);
    Quad outlineQuad(colorShader);

    glEnable(GL_DEPTH_TEST);
    glDepthFunc(GL_LESS);
    glEnable(GL_STENCIL_TEST);
    glStencilFunc(GL_NOTEQUAL, 1, 0xFF);
    glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);

    bool quit = false;

    while(!quit) {
        glClearColor(0.25f, 0.25f, 0.25f, 1.0f);
        glClear(GL_COLOR_BUFFER_BIT | GL_STENCIL_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

        glDisable(GL_RASTERIZER_DISCARD);
        glStencilMask(0x00);

        //binds everything, and call glDraw*, its parameter is model matrix and u_r uniform
        //it is also tested and works fine
        sceneQuad->Draw(Scale(1.0f), 1.0f);

        glEnable(GL_RASTERIZER_DISCARD);

        glStencilFunc(GL_ALWAYS, 1, 0xFF);
        glStencilMask(0xFF);
        maskQuad->Draw(); // it does not need matrix or u_r, because it uses different shader

        glDisable(GL_RASTERIZER_DISCARD);

        glStencilFunc(GL_NOTEQUAL, 1, 0xFF);
        glStencilMask(0x00);
        glDisable(GL_DEPTH_TEST);

        oulineQuad->Draw(Scale(1.1), 0.0f); //a bit scaled 
    }
}

color shader:

#version 460 core

layout (location = 0) in vec2 a_position;

uniform mat4 u_model;

void main()
{
    gl_Position = u_model * vec4(a_position, 0.0f, 1.0f);
}

//-------------------------------------------------------------------

#version 460 core

layout (location = 0) out vec4 f_color;

uniform float u_r;

void main()
{        
    f_color = vec4(u_r, 0.0f, 1.0f, 1.0f);
}

mask shader, just vertex program:

#version 460 core

layout (location = 0) in vec2 a_position;

void main()
{
    gl_Position = vec4(a_position, 0.0f, 1.0f);
}

what I expect is to draw the purple quad, then draw it again but without fragment shader, only drawing to stencil buffer and then draw the blue quad on top of everything but discarding pixels that are drawn by mask in stencil buffer.

what it does, it draws the blue quad without discarding mask pixels.

how to fix it?

Answer 1

From "What actually does GL_RASTERIZE_DISCARD":

When enabled, primitives are discarded immediately before the rasterization stage, but after the optional transform feedback stage (see section 13.2)

Given that you want to draw to the stencil buffer, you still want rasterization, just not to the color buffer .
For that, you can either use

glDrawBuffer(GL_NONE)

or

glColorMask(GL_FALSE,GL_FALSE,GL_FALSE,GL_FALSE)

See this post for a comparison, but either way is fine for your application I think.