I am using python's cgkit Mat44.lookat method and I am wondering if it is implemented correctly. I wrote the author but got no answer so far. My question is maybe silly, but I would really like to understand what's wrong in my program.
if I do :
camera_to_world_matrix = mat4.lookAt(eye=vec3(0,0,1), target=vec3(0,0,-3), up=vec3(0,1,0))
world_to_camera_matrix = camera_to_world_matrix.inverse()
pos_in_camera_space = world_to_camera_matrix * vec3(0,0,-10)
In my opinion, pos_in_camera_space should equal to vec3(0,0,-11) : the distance is 11 units between the two positions and as we are in a right handed system, the camera look toward the negative part of the z axis.
But I get vec3(0,0,11) and I don't see why.
edit : I use OpenGl3 and therefore shaders.
The handedness is actually resolved by your projection matrix in the very end. You can invert the handedness of all of your transformations merely by making (0,0) the top-left corner of your screen instead of bottom-left for instance (glOrtho (0, width, 0, height, -1, 1) vs. glOrtho (0, width, height, 0, -1, 1)).
The reason gluLookAt (...) produces a right-handed transformation is because the matrix functions in GL that are used to establish the projection matrix (e.g. glOrtho (...) and glFrustum (...)) invert the Z coordinate. It is only when you combine the two matrices that you have a right-handed view space. Since they are inseparable in fixed-function OpenGL this point goes largely unnoticed.
For a more thorough explanation with diagrams, see this related question: Is OpenGL coordinate system left-handed or right-handed?