Search code examples
openglfragment-shaderstencil-buffer

How does gl_FragColor affect the stencil buffer?

I have set up my operations to draw to the stencil buffer, similar to the following:

void onDisplay() {
  glClear(GL_DEPTH_BUFFER_BIT);
  glEnable(GL_STENCIL_TEST);
  glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
  glDepthMask(GL_FALSE);
  glStencilFunc(GL_NEVER, 1, 0xFF);
  glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP);

  // Draw to stencil buffer
  glStencilMask(0xFF);
  glClear(GL_STENCIL_BUFFER_BIT);  // needs mask=0xFF
  draw_circle();

  glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
  glDepthMask(GL_TRUE);
  glStencilMask(0x00);

  // draw where stencil's value is 0
  glStencilFunc(GL_EQUAL, 0, 0xFF);
  /* (nothing to draw) */
  // draw only where stencil's value is 1
  glStencilFunc(GL_EQUAL, 1, 0xFF);

  draw_scene();

  glDisable(GL_STENCIL_TEST);
}

Now, if I have the following fragment shader enabled when I call draw_circle() (above):

void main() {
  gl_FragColor = vec4(1.0, 0.0, 0.0, 1.0);
}

How will the values of the stencil buffer differ from if I were to use the following fragment shader?

void main() {
  gl_FragColor = vec4(0.0, 1.0, 0.0, 1.0);
}

In other words, I'm wondering how the output of the fragment shader affects the stencil buffer when drawing to the stencil buffer.

Edit:

The point of my question is to correct some misunderstandings I know I have regarding the stencil buffer. One example I have that I think explains the stencil buffer fairly well is [1]. Here, the following is mentioned:

The glColorMask function allows you to specify which data is written to the color buffer during a drawing operation. In this case you would want to disable all color channels (red, green, blue, alpha). Writing to the depth buffer needs to be disabled separately as well with glDepthMask, so that cube drawing operation won't be affected by leftover depth values of the rectangle. This is cleaner than simply clearing the depth buffer again later.

So, it seems from this page, that, in order to write to the stencil buffer, one needs to enable/disable the appropriate modes (i.e. color and depth), and then go through the entire rasterization process, which will only write to the stencil buffer. Since the rasterization process includes the fragment shader, is the output of the fragment shader (i.e. gl_FragColor) simply ignored? How can I tell GL what to write to the stencil buffer position (x, y)?

[1] : https://open.gl/depthstencils

Solution

  • Unless you have access to AMD/ARB_shader_stencil_export, the fragment shader cannot directly affect the output to the stencil buffer. The only exception to this is discarding the fragment.

    And according to this database, only AMD cards provide this extension. Also, that extension exposes an output specifically for the stencil. It modifies the stencil value of the fragment; the color values of the fragment never affect the fragment's stencil value.