I have my own unproject function for performing reverse projection of a screen point. The code is as follows (written in OpenTK):
public static Vector3 UnProject(Point screenLocation, float depth)
{
int[] viewport = GetViewport();
Vector4 pos = new Vector4();
// Map x and y from window coordinates, map to range -1 to 1
pos.X = (screenLocation.X - viewport[0]) / (float)viewport[2] * 2.0f - 1.0f;
pos.Y = 1 - (screenLocation.Y - viewport[1]) / (float)viewport[3] * 2.0f;
pos.Z = depth * 2.0f - 1.0f;
pos.W = 1.0f;
Vector4 pos2 = Vector4.Transform(pos, Matrix4.Invert(GetModelViewMatrix() * GetProjectionMatrix()));
Vector3 pos_out = new Vector3(pos2.X, pos2.Y, pos2.Z);
return pos_out / pos2.W;
}
Basically, you'd provide the desired unprojection depth to my function, and it will give you the corresponding world coordinate of the screen point. Assuming that this works correctly (which I am 99% sure think it does), I'm having problems converting screen points to world coordinates. This unprojection works fine for picking: I'd call my unproject function twice (once with depth = 0 and another time with depth = 1) to convert the screen point to ray. I perform ray/triangle intersection to determine which object intersects with the ray and based on that I perform picking (which works very accurately).
For another operation (let's call it operation X), I only need to know the world coordinate of the screen point (assuming that the mouse cursor is over an object on the screen). For that, I am obtaining the depth under the cursor by using the glReadPixel function. The problem is that I feel the Z value obtained by reading the depth buffer is a little bit off. If I calculate the intersection with ray casting, I get accurate results, but that is not viable for operation X as operation X needs to be performed every time MouseMoved is triggered.
To demonstrate the lack of accuracy, here are the two numbers I obtained:
glReadPixel + Unprojection yields (0.886105343709181, 0.12422376198582, 0.998496665566841) as the world coordinate under the cursor.
Ray casting + intersection yields (0.885407337013061, 0.124174778008613, 1) as the world coordinate under the cursor.
This 0.0015 error in the Z value is too much for operation X (as it is very sensitive to small numbers).
Is there something wrong with glReadPixels that I should know about? Is this happening because glReadPixels is only capable of reading float values?
I don't think that glReadPixels is to blame here. I think that the Z buffer precision is the issue. By default, you typically have a 24 bit fixed-point depth buffer. Maybe it helps if you use a 32 bit floating point depth buffer, but you probably need an FBO for that.