I'm trying to convert my mouse's cursor position in my OpenGL viewport to world coordinates. I'm using glm::unProject() to do this. However, it appears that the mouse position's Y coordinate is being negated somehow.
If I orient my camera so the world's Y axis is pointing up and X is pointing right, moving the mouse left/right gives me the correct coordinates, however moving the mouse up/down the Y "world" coordinates I get are reversed (positive Y goes downwards).
If I reorient the camera so X is now going up, Y is going left. Moving the mouse left/right gives the right Y coordinates, but moving up/down gives reversed X coordinates. Same behavior when I orient for Z.
This page mentions that device coordinates use the LHS, maybe this is the cause? Is there something I need to do to handle the case where device coordinates are in a different system? Is there a way to determine that?
I'm also noticing that my transformed coordinates are half what they should be (mouse on an object at (1,0,0) shows (0.5,0,0) but I think this is a separate issue so I'll ask another question once I solve this.
The basic problem is that OpenGL defines the origin as the lower left corner of the window, while most windowing systems use the upper left instead. The solution is simple: subtract the mouse Y coordinate from the window height:
gl_x = mouse_x;
gl_y = windowHeight - mouse_y;