libgdx open gles 2.0 stencil alpha masking

Question

I'm looking for a solution to implement alpha masking with stencil buffer in libgdx with open gles 2.0.

I have managed to implement simple alpha masking with stencil buffer and shaders, where if alpha channel of fragment is greater then some specified value it gets discarted. That works fine.

The problem is when I want to use some gradient image mask, or fethered png mask, I don't get what I wanned (I get "filled" rectangle mask with no alpha channel), instead I want smooth fade out mask.

I know that the problem is that in stencil buffer there are only 0s and 1s, but I want to write to stencil some other values, that represent actual alpha value of fragment that passed in fragment shader, and to use that value from stencil to somehow do some blending. I hope that I've explained what I want to get, actually if it's possible.

I've recently started playing with OpenGL ES, so I still have some misunderstandings.

My questions is: How to setup and stencil buffer to store values other then 0s and 1s, and how to use that values later for alpha masking?

Tnx in advance.

This is currently my stencil setup:

    Gdx.gl.glClearColor(1, 1, 1, 1);
    Gdx.gl.glClear(GL20.GL_COLOR_BUFFER_BIT | GL20.GL_STENCIL_BUFFER_BIT |   GL20.GL_DEPTH_BUFFER_BIT);

    // setup drawing to stencil buffer
    Gdx.gl20.glEnable(GL20.GL_STENCIL_TEST);
    Gdx.gl20.glStencilFunc(GL20.GL_ALWAYS, 0x1, 0xffffffff);
    Gdx.gl20.glStencilOp(GL20.GL_REPLACE, GL20.GL_REPLACE, GL20.GL_REPLACE);
    Gdx.gl20.glColorMask(false, false, false, false);
    Gdx.gl20.glDepthMask(false);

    spriteBatch.setShader(shaderStencilMask);
    spriteBatch.begin();

    // push to the batch
    spriteBatch.draw(Assets.instance.actor1, Gdx.graphics.getWidth() / 2, Gdx.graphics.getHeight() / 2, Assets.instance.actor1.getRegionWidth(), Assets.instance.actor1.getRegionHeight());

    spriteBatch.end();

    // fix stencil buffer, enable color buffer
    Gdx.gl20.glColorMask(true, true, true, true);
    Gdx.gl20.glDepthMask(true);
    Gdx.gl20.glStencilOp(GL20.GL_KEEP, GL20.GL_KEEP, GL20.GL_KEEP);

    // draw where pattern has NOT been drawn
    Gdx.gl20.glStencilFunc(GL20.GL_EQUAL, 0x1, 0xff);

    decalBatch.add(decal);
    decalBatch.flush();

    Gdx.gl20.glDisable(GL20.GL_STENCIL_TEST);

    decalBatch.add(decal2);
    decalBatch.flush();

Answer 1

The only ways I can think of doing this are with a FrameBuffer.

Option 1

Draw your scene's background (the stuff that will not be masked) to a FrameBuffer. Then draw your entire scene without masks to the screen. Then draw your mask decals to the screen using the FrameBuffer's color attachment. Downside to this method is that in OpenGL ES 2.0 on Android, a FrameBuffer can have RGBA4444, not RGBA8888, so there will be visible seams along the edges of the masks where the color bit depth changes.

Option 2

Draw you mask decals as B&W opaque to your FrameBuffer. Then draw your background to the screen. When you draw anything that can be masked, draw it with multi-texturing, multiplying by the FrameBuffer's color texture. Potential downside is that absolutely anything that can be masked must be drawn multi-textured with a custom shader. But if you're just using decals, then this isn't really any more complicated than Option 1.

The following is untested...might require a bit of debugging.

In both options, I would subclass CameraGroupStrategy to be used with the DecalBatch when drawing the mask decals, and override beforeGroups to also set the second texture.

public class MaskingGroupStrategy extends CameraGroupStrategy{

    private Texture fboTexture;

    //call this before using the DecalBatch for drawing mask decals
    public void setFBOTexture(Texture fboTexture){
        this.fboTexture = fboTexture;
    }

    @Override
    public void beforeGroups () {
        super.beforeGroups();
        fboTexture.bind(1);
        shader.setUniformi("u_fboTexture", 1);
        shader.setUniformf("u_screenDimensions", Gdx.graphics.getWidth(), Gdx.graphics.getHeight());
    }
}

And in your shader, you can get the FBO texture color like this:

vec4 fboColor = texture2D(u_fboTexture, gl_FragCoord.xy/u_screenDimensions.xy);

Then for option 1:

gl_FragColor = vec4(fboColor.rgb, 1.0-texture2D(u_texture, v_texCoords).a);

or for option 2:

gl_FragColor = v_color * texture2D(u_texture, v_texCoords);
gl_FragColor.a *= fboColor.r;