I was trying to determine if an object (sphere) is inside a view frustum. My strategy was first to get the view matrix:
glm::lookAt(cameraPosition, lookAtPosition, cameraUp);
Where cameraPosition is the position of the camera, lookAtPosition is calculate by cameraPosition + cameraDirection and cameraUp is pretty much self-explanatory.
After obtaining the view matrix, I calculate the 8 vertices of the frustum in the camera view, then multiply the inverse of the view matrix by each point to get the coordinate of them in the world space.
Using these 8 points in the world space, I can construct the 6 planes by using cross products (so their normals will point inward). Then I take the dot product of the object vector (I get the vector by subtracting the object position with an arbitrary point on that plane) with the normal vector of each plane, if it is positive for all 6, I know it's inside the frustum.
So my questions are:
My frustum is opposite with my cameraDirection which causes nothing to be seen on the screen (but it still drawing stuff in the back of my camera).
Is the camera view is always set at (0,0,0) and looking at the positive z direction?
The view space is the space which defines the look at the scene.
Which part is "seen" (is projected onto the viewport) is depends on the projection matrix. Generally (in OpenGL) the origin is (0,0,0) and the view space z axis points out of the viewport. However, the projection matrix inverts the z axis. It turns from a the Right-handed system of the view space to the left handed system of the normalized device space.
the inverse of it to transform the camera view to the world view
Yes.
The view matrix transforms from world space to view space. The projection matrix transform the Cartesian coordinates of the view space to the Homogeneous coordinates of the clipspace. Clipspace coordinates are transformed to normalized device space by dividing the xyz components by the w component (Perspective divide).
The normalized device space is a cube with a minimum of (-1, -1, -1) and a maximum of (1, 1, 1). So the 8 corner points of the cube are the corner of the viewing volume in normalized device space.
(-1, -1, -1) ( 1, -1, -1) ( 1, 1, -1) (-1, 1, -1) // near plane
(-1, -1, 1) ( 1, -1, 1) ( 1, 1, 1) (-1, 1, 1) // far plane
If you want to transform a point from normalized device space to view space, you've to:
glm::mat4 view; // view matrix
glm::mat4 projetion; // projection matrix
glm::vec3 p_ndc; // a point in normalized device space
glm::mat4 inv_projetion = glm::inverse( projetion );
glm::mat4 inv_view = glm::inverse( view );
glm::vec4 pth = inv_view * inv_projetion * glm::vec4( p_ndc, 1.0f );
glm::vec3 pt_world = glm::vec3( pth ) / pth.w;