GLSL access coverage mask of sampler2D texel

Question

If I access a multisample texture in GLSL through a sampler2DMS, how do I know which of the samples in a texel of this multisample texture have actually been covered?

From the multisample extension reference: "... Each pixel fragment thus consists of integer x and y grid coordinates, a color, SAMPLES_ARB depth values, texture coordinates, and a coverage value with a maximum of SAMPLES_ARB bits."

So what I would like to access is the coverage value of the texel. There is gl_SampleMask (https://www.opengl.org/sdk/docs/man/html/gl_SampleMask.xhtml) that I can use to WRITE the coverage value of the FRAGMENT currently processed, but how do I access the coverage value of the TEXEL I'm fetching from the multisample texture?

Answer 1

The idea with multisampling is that, when you render to a multisampled image, you only execute the fragment shader once for each pixel-sized area. The rasterizer-generated coverage mask determines which samples within the pixel the fragment's outputs go to.

But once that process is done, once the fragment shader writes its data, the multisample image itself has absolutely no idea what these coverage masks are. A multisample texture simply has multiple sample values per texel. It has no idea what fragments generated which samples with which sample masks.

Sample masks are only a part of rendering.

Think of it like this. This is a multisample texture's pixel:

vec4 pixel[SAMPLE_COUNT];

Your fragment shader, when you were rendering to the multisample texture, did the equivalent of this:

for(int sample_ix = 0; sample_ix < SAMPLE_COUNT; ++sample_ix)
{
  if(sampleMask[sample_ix])
    pixel[sample_ix] = output;
}

pixel's data may have originally come from a sample mask. But pixel has no idea that this happens; it's just an array of vec4 values.

You can get the coverage value of your current fragment. But that has no relation to the actual coverage value(s) used to originally compose the pixels in a multisample texture.