Tracking objects in a 3D environment with 2D HUD icons

Question

I am attempting to have a 2D HUD which has icons that track the location on the screen of 3D objects behind the HUD in a 3D environment.

Reasoning: Sometimes you cannot see a 3D object (too far away or off screen) but you still want to know where it is.

Issue: 3D scene is using a perspective matrix to transform it, giving it depth (z-axis), the HUD is strictly 2D (xy-plane). Because of the depth, the 2D HUD cannot properly track objects when they are farther/closer away.

What I want: A way to get a 2D Vector [(x,y) pos] of where to put an icon so that it is centered where the 3D object in the background would be.

Example of all objects in an xy-plane (z=0):

You can see that as the objects get farther away from the center, the Icon (circle thing in white) is more off center.

Example of objects with increasing depths (farther from center == deeper):

You can see that the HUD thinks 3D objects are in the same plane still.

Pseudo-Code:

.getPos() gets the Vector (x,y,z)

lookAtObj = Object.getPos() - camera.getPos() // lookAt vector from camera to the object
icon.pos = Orthogonal Component of lookAtObj on camera.get_lookAt()

My Perspective Matrix:

// Function call in the OpenGL draw() method
FloatMatrix proj = FloatMatrix.getPerspectiveMatrix( this.fov, this.width, this.height, 0.1f, 200.0f );

// Function
public static FloatMatrix getPerspectiveMatrix( Double fov, float w, float h, float near, float far ){
    float asp = w/h;
    float fov_cos = (float) Math.cos( fov / 2.0d );
    float fov_sin = (float) Math.sin( fov / 2.0d );
    float fov_cot = fov_cos/fov_sin;
    float a_0  = fov_cot/asp;
    float a_3  = (far + near)/(near-far);
    float a_43 = (2.0f * far * near)/(near-far);
    float[] an = {
            a_0,  0.0f,    0.0f, 0.0f,
            0.0f, fov_cot, 0.0f, 0.0f,
            0.0f, 0.0f,    a_3,  -1.0f,
            0.0f, 0.0f,    a_43, 0.0f,
    };
    return new FloatMatrix( an, 4, 4 );

}

Answer 1

This is pretty straightforward. You can use gluProject. It will take a given modelview, projection, and viewport transform, and a 3D point, and apply the inverse and spit out a 2D point in window coordinates for you (apologies for minor typos, just typing this here):

double myX = ..., myY = ..., myZ = ...; // your object's 3d coordinates
double[] my2DPoint = new double[2]; // will contain 2d window coords when done

double[] modelview = new double[16];
double[] projection = new double[16];
int[] viewport = new int[4];

gl.glGetDoublev(GL2.GL_MODELVIEW_MATRIX, modelview, 0);
gl.glGetDoublev(GL2.GL_PROJECTION_MATRIX, projection, 0);
gl.glGetIntegerv(GL2.GL_VIEWPORT, viewport, 0);
glu.gluProject(myX, myY, myZ, modelview, 0, projection, 0,
               viewport, 0, my2DPoint, 0);

// now my2DPoint[0] is window x, and my2DPoint[1] is window y

After you do this, you'll have your 3D point in 2D window coordinates. Then simply switch your projection over to a 2D orthogonal projection, in window pixels, and draw your HUD in 2D space.

For performance, if you have multiple HUD items to draw per frame; just get the modelview/projection/viewport once per frame (or, even better, invalidate your cached ones if you change them and re-query only as needed) and reuse them in subsequent calls to gluProject.