how to return the mask of sampled pixels

Question

I am using a program, which is available on GitHub (here), to convert an equirectangular image (360-degree images) to a perspective view (viewport) using OpenGL. For that, the ray-tracing approach is used to find the intersection of fragment points on the unit sphere and later this information will be used to sample required pixels from equirectangular images and project them on perspective view. Everything works fine, but now along the projection result, I need the program to return to me also a mask image (of type unsigned byte and same size of the equirectangular image) which represents pixels which are sampled from the equirectangular images.

How can I change the fragment shader to return to me this unsigned byte mask (0 represents non-samples and 255 represents sampled pixels) at the same time that it is rendering the view on frame buffer?

To be more specific, my current fragment shader looks as follows:

const float pi = 3.141592653589793;
varying vec3 planePoints;
uniform sampler2D tex;
void main()
{
        vec2 equirectangularTexturePos;
        vec3 spherepos = normalize(planePoints);
        equirectangularTexturePos.x = (atan(spherepos.y, -spherepos.x)+pi)/(2.0*pi);
        equirectangularTexturePos.y = acos(spherepos.z)/(pi);
        gl_FragColor = texture2D(tex, equirectangularTexturePos);
}

and what I want is something similar as follows be added to my fragment shader:

texture2D(mask, equirectangularTexturePos) = 255;

Answer 1

I am just writing this to elaborate on a possible solution which I tried to hint to in the above comment using Image Load Store. Firstly, you need to create a new texture of the same dimension of your equirectangular texture (you can use GL_R8 as internal format). You can either do it with glTexImage2D or glTexStorage2D.

Then you need to declare an image binding point in your shader and bind the first level of your mask image to that image binding point. Then, in your shader, when you sampled from your original color texture you are also going to use the imageStore() function to write to the mask texture at that texel location. Attention: The texel location is in actual texels and not in a fraction of the whole texture dimension!

For a first approximation we just use the truncated integer coordinate of your original sample position (for your texture2D call). Actually, since you use GL_LINEAR as filtering function, you should compute the nearest 4 texels and write to those. See PDF page 150 of https://www.khronos.org/registry/OpenGL/specs/gl/glspec13.pdf to know which algorithm the GL uses to determine the four texels for LINEAR filtering.

So parts of your shader are then going to look like this:

#extension GL_ARB_shader_image_load_store : enable

layout(binding = 0, r8) writeonly uniform image2D maskImage;
...
ivec2 texel = ivec2(equirectangularTexturePos * textureSize(tex, 0));
imageStore(maskImage, texel, vec4(1.0, 0.0, 0.0, 0.0));

In order to bind an image to an image binding point, you would use this in your host program:

glBindImageTexture(0, maskTex, 0, false, 0, GL_WRITE_ONLY, GL_R8);

Also do not forget to clear the texture before running the shader. After the shader ran and filled the accessed texels of level 0 of your mask texture, you need to add an appropriate synchronization barrier, so that further GL calls sourcing from that texture see the updated texture values. You do that via:

glMemoryBarrier(GL_SHADER_IMAGE_ACCESS_BARRIER_BIT);

Now you can use that texture in any way and source from it for example with another shader.