How to use depth buffer to clip table in libgdx's android game

Question

Table' default clip is to use scissorstack,but scissorstack support only rectangle. And I want to clip it with circle ,so I choose to use depth buffer.

Here are my code,but it don't clip table successfully. the table displayed as what it used to be.

@Override
protected void drawBackground(Batch batch, float parentAlpha, float x, float y) {
    // TODO Auto-generated method stub
                 batch.end();
               //       //1. clear screen
                 Gdx.gl.glClear(GL10.GL_COLOR_BUFFER_BIT);

                //2. clear our depth buffer with 1.0
                Gdx.gl.glClearDepthf(1f);
                Gdx.gl.glClear(GL10.GL_COLOR_BUFFER_BIT);

                //3. set the function to LESS
                Gdx.gl.glDepthFunc(GL10.GL_LESS);

                //4. enable depth writing
                Gdx.gl.glEnable(GL10.GL_DEPTH_TEST);

                 //5. Enable depth writing, disable RGBA color writing 
                Gdx.gl.glDepthMask(true);
                Gdx.gl.glColorMask(false, false, false, false);

                ///////////// Draw mask shape(s)

                //6. render your primitive shapes
                shapes.begin(ShapeType.Filled);

                shapes.setColor(1f, 1f, 1f, 0.5f);
                shapes.circle( PositionArray.detail_x, PositionArray.detail_y,100);

                shapes.end();
                batch.begin();              

                ///////////// Draw sprite(s) to be masked
                //8. Enable RGBA color writing
                //   (SpriteBatch.begin() will disable depth mask)
                Gdx.gl.glColorMask(true, true, true, true);

                 //9. Make sure testing is enabled.
                Gdx.gl.glEnable(GL10.GL_DEPTH_TEST);

                //10. Now depth discards pixels outside our masked shapes
                Gdx.gl.glDepthFunc(GL10.GL_EQUAL);

                //push to the batch
                super.drawBackground(batch, parentAlpha, x, y);
                //  batch.draw(grass, 0, 0);
                //end/flush your batch
                Gdx.gl.glDisable(GL10.GL_DEPTH_TEST);
}   

Any help will be appreciate!

Improvement:

After adopting Tenfour04' advice,I change my code ,but it still don't solve my problem.

public class MyTable extends Table {
ShapeRenderer shapes;
private final Matrix4 tmpM = new Matrix4();
    public MyTable() {
        super();
        // TODO Auto-generated constructor stub
        shapes = new ShapeRenderer();
    }
@Override
protected void drawBackground(Batch batch, float parentAlpha, float x, float y) {
    // TODO Auto-generated method stub
                 batch.end();
               //       //1. clear screen
//Gdx.gl.glClear(GL10.GL_COLOR_BUFFER_BIT);
//                  
////2. clear our depth buffer with 1.0
//Gdx.gl.glClearDepthf(1f);
//Gdx.gl.glClear(GL10.GL_COLOR_BUFFER_BIT);


                //3. set the function to LESS
                Gdx.gl.glDepthFunc(GL10.GL_LESS);

                //4. enable depth writing
                Gdx.gl.glEnable(GL10.GL_DEPTH_TEST);

                 //5. Enable depth writing, disable RGBA color writing 
                Gdx.gl.glDepthMask(true);
                Gdx.gl.glColorMask(false, false, false, false);

                ///////////// Draw mask shape(s)

                //6. render your primitive shapes
                shapes.setProjectionMatrix(getStage().getViewport().getCamera().combined);
                shapes.setTransformMatrix(tmpM.idt().translate(0, 0, -0.5f));
                shapes.begin(ShapeType.Filled);

                shapes.setColor(1f, 0f, 0f, 0.5f);
                shapes.circle( PositionArray.detail_x, PositionArray.detail_y,100);

                shapes.end();

                batch.begin();              

                ///////////// Draw sprite(s) to be masked
                //8. Enable RGBA color writing
                //   (SpriteBatch.begin() will disable depth mask)
                Gdx.gl.glColorMask(true, true, true, true);

                 //9. Make sure testing is enabled.
                Gdx.gl.glEnable(GL10.GL_DEPTH_TEST);

                //10. Now depth discards pixels outside our masked shapes
                Gdx.gl.glDepthFunc(GL10.GL_LEQUAL);

                //push to the batch
                super.drawBackground(batch, parentAlpha, x, y);
                //  batch.draw(grass, 0, 0);
                //end/flush your batch
                batch.flush();
                Gdx.gl.glDisable(GL10.GL_DEPTH_TEST);
   }                    
}

Answer 1

I see a few issues:

1) You disable depth testing before the batch is flushed. SpriteBatch only sends meshes to the GPU for drawing when it gets flushed or when you call end on it. So add the line batch.flush() before you disable depth testing.

2) You never cleared the depth buffer bit. Also, you should not be calling glClear more than once. Preferably, you would be calling it before stage.draw() not in the middle of an actor's draw method! I would put the following before stage.draw():

Gdx.gl.glClear(GL20.GL_COLOR_BUFFER_BIT | GL20.GL_DEPTH_BUFFER_BIT);

and remove the similar lines from your Actor. There's no reason to call Gdx.gl.glClearDepthf(1f); since the default is 1 anyway.

3) If your mask shape is in the same units as everything else you're drawing, the shape render needs to use the same projection matrix as the stage. Also, by leaving the matrix at the default, I think you are drawing your mask at Z=1 (depth 0), not at Z=0, which is where sprite batch is drawing. Put this before shapes.begin:

shapes.setProjectionMatrix(getStage().getViewport().getCamera().combined);

4) I suppose it's possible that rounding could cause the GL_EQUAL depthFunc to fail. If you do all the above and it still doesn't work, try drawing the circle farther back.

private final Matrix4() tmpM = new Matrix4();

//...

//right before shapes.begin:
shapes.setTransformMatrix(tmpM.idt().translate(0, 0, -0.5f));

And then when you set the depthFunc for the masked sprites, use LEQUAL instead of EQUAL.