Intersection of mouse cursor and projected 2D rectangle

Question

There is a bunch of 2x2 "floor/tiles" in a 3D-world with JOGL, they are projected into the camera and I want to know the cursor is hovering the tile or not.

I have a camera settings like this:

glu.gluPerspective(90, 4.0/3.0, 1, 100);
glu.gluLookAt(-2f, 8f, -2f, 0f, 0f, 0f, 0f, 1f, 0f);

and there are some tiles or blocks on the y=0 pane like this

gl.glPushMatrix();
{
    // Camera settings (same as above)
    glu.gluPerspective(90, 4.0/3.0, 1, 100);
    glu.gluLookAt(-2f, 8f, -2f, 0f, 0f, 0f, 0f, 1f, 0f);

    // Draw the tiles
    gl.glPushMatrix();
    {
        gl.glBegin(GL2.GL_POLYGON);
        {
            // a bunch of translated and textured
            // (1,0,1) (1,0,-1) (-1,0,-1) (-1,0,1)
            // rectangle here, 
        }
        gl.glEnd();
    }
    gl.glPopMatrix();
}
gl.glPopMatrix();

I am new in 3D and I am only familiar with Java Graphics2D. Intersection of 2D rectangle and cursor is just a few easy comparison, but it seems to be a lot more complicated in 3D. I am looking for some Maths or library to do this.

Or if there is a method to get the 4 point of the final pixels on the screen, Maybe I would like to do java.awt.Shape contain() and check it intersects or not.

The result will be like this:

Answer 1

Maybe the simplest solution is using the gluProject(); to get the screen coordinate, and using java.awt.geom.Path2D to check the mouse coordinate is in the area or not.

Here is a simple sample code:

// do inside the matrix stack? of rendering rectangles to get the right matrix we want
float[][] FinalCoordinate = new float[4][4];
float[] ModelView = new float[16];
float[] Projection = new float[16];
gl.glGetFloatv(GL2.GL_MODELVIEW_MATRIX,ModelView,0);
gl.glGetFloatv(GL2.GL_PROJECTION_MATRIX,Projection,0);

for(int x = 0; x < 4; x++)
{
    glu.gluProject(Vertex[x][0],0f,Vertex[x][2],ModelView,0,Projection,0,new int[]{0,0,800,600},0,FinalCoordinate[x],0);;
}

after getting four points of the final coorindates, use Path2D to check the intersection:

Path2D p = new Path2D.Float();
p.moveTo(FinalCoordinate[0][0],FinalCoordinate[0][1]);
for(int x = 1;x<4;k++)
{
    p.lineTo(fc[x][0],fc[x][1]);
}
p.closePath();

boolean Result = p.contains(MouseX, MouseY);

thats it! Thanks those suggestions and links :)