Alpha mask of overlapping objects in OpenGL

Question

What is the best way of getting alpha mask of overlapping objects in OpenGL? In the first picture below I have three grids overlapping with ellipsoid, depth test is enabled. My goal is to get result similar to the second image, where the white represents the alpha. Below are the depth testing flags that I am using.

glEnable(GL_DEPTH_TEST);
glDepthMask(GL_TRUE);
glDepthFunc(GL_LEQUAL);

Answer 1

Before you draw your red, yellow and blue grids you should enable the stencil test and set your stencil operations glStencilOp function like this:

glEnable( GL_STENCIL_TEST );
glStencilOp( GL_KEEP, GL_KEEP, GL_INCR );
glStencilFunc( GL_ALWAYS, 0, 1 ); // these are also the default parameters

This means the stencil buffer is kept as is is if the depth test fails and increment if the depth test passes. After drawing the stencil buffer contains the mask, where 0 means black and > 0 means white in your case. Ensure your stencil buffer to be cleared before drawing ( glClear( GL_STENCIL_BUFFER_BIT ) ).

If you have to get this mask to a texture you have to bind a texture to the stencil buffer:

GLint with = ...;
GLint height = ...;

GLuint depthAndStencilMap;
glGenTextures( 1, &depthAndStencilMap );
glBindTexture( GL_TEXTURE_2D, depthAndStencilMap );
glTexImage2D( GL_TEXTURE_2D, 0, GL_DEPTH_STENCIL, with, height, 0, GL_DEPTH_STENCIL, GL_FLOAT, NULL );
glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST );
glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST );
glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_REPEAT );
glTexParameteri( GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_REPEAT );

GLuint frameBuffer;
glGenFramebuffers( 1, &frameBuffer );
glBindFramebuffer( GL_FRAMEBUFFER, frameBuffer );

glFramebufferTexture2D( GL_FRAMEBUFFER, GL_DEPTH_STENCIL_ATTACHMENT, GL_TEXTURE_2D, depthAndStencilMap, 0 );

glBindFramebuffer( GL_FRAMEBUFFER, 0);

Before drawing you have to bind and clear the frame buffer:

glBindFramebuffer( GL_FRAMEBUFFER, frameBuffer );
glClear( GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT )

After drawing the texture depthAndStencilMap contains the depth buffer in the Red channel and the stencil buffer, which is your mask, in the 'Blue' channel. Note the scene is drawn to the frame buffer and not further to the viewport.