Get world coordinates of frustum from matrix

Question

I have the usual cube (-1, -1, -1 to 1, 1, 1) and my camera matrices (view and projection). The projection matrix is a perspective projection. I want to see which area is visible from the given camera. To visualize that I want to stretch my cube like the frustum, so that I can see the result from a other camera.

My Idea:

Get the inverse of the viewProjectionMatrix and multiply the unit cube coordinates with it:

glm::vec4 CubeCoordinates[8]{                                        
    glm::vec4(-1.0f, -1.0f, 1.0f, 1.0f),
    glm::vec4(-1.0f, 1.0f, 1.0f, 1.0f), 
    glm::vec4(1.0f, 1.0f, 1.0f, 1.0f), 
    glm::vec4(1.0f, -1.0f, 1.0f, 1.0f), 
    glm::vec4(-1.0f, -1.0f, -1.0f, 1.0f), 
    glm::vec4(-1.0f, 1.0f, -1.0f, 1.0f), 
    glm::vec4(1.0f, 1.0f, -1.0f, 1.0f), 
    glm::vec4(1.0f, -1.0f, -1.0f, 1.0f) };

glm::mat4 InvertedViewProjectionMatrix = glm::inverse(refCamera.GetViewProjectionMatrix());

for (uint32_t i = 0; i < 8; ++i)
{
    CameraEdgesWorldSpace[i] = InvertedViewProjectionMatrix * CubeCoordinates[i];
}

Now the cube coordinates should be in worldspace and look like the frustum. But the outcoming coordinates are wrong.

I can guarantee, that the viewProjectionMatrix is correctly working, because I use it like everywhere.

Is the calculation wrong? or does I use the wrong command to invert the matrix (I'm new to GLM)?

Answer 1

The projection matrix maps eye coordinates to clip coordinates. Now, you don't have clip coordinates to back-project, you only have the NDC ones.

How do you convert from NDC to clip coordinates? Well, keep in mind we are working with homogeneous rather than Euclidean coordinates (hence the 4th dimension w). The homogeneous coordinates happen to be invariant with respect to multiplication by a non-zero constant. That is, point (a, b, c, d) is equivalent to (k*a, k*b, k*c, k*d) for any non-zero k.

You go from clip coordinates to NDC by multiplying them by 1/w. That's a multiplication by a constant! Therefore, NDC is by itself a valid clip coordinate, so you can back-project that.

Now, going from homogeneous to Euclidean coordinates always requires dividing by w. The reason we normally don't do it in OpenGL is that affine transformations don't affect w, so it usually remains 1. Not in this case though, as both the projection matrix and its inverse do affect the w coordinate.

Therefore, as pointed out by BDL in comments, you are missing a division by w. You need to take the visible cube in NDC coordinates, pass them through a chain of backwards transformations, and finally divide by w in order to go to Euclidean coordinates. In fact, you may well skip the division by w, as you may simply stay in homogeneous coordinates and render that with a view from another camera.