I am trying to write a method that calculates the view frustum corners for a camera class I am working on. I know that the MVP matrix is correct because it is pushed into my vertex shader and indeed everything is correct on screen, not only that but I calculate the worldspace frustum planes from this matrix and they are correct. Also those methods are unit tested in a few different camera configurations and give expected results.
I am attempting to calculate the frustum corners by transforming the NDC corners by the inverse of the MVP matrix.
// Fill with the corners in clip space.
QVector< Vector3f > corners( 8 );
corners[0] << 1.0f, 1.0f, 1.0f;
corners[1] << -1.0f, 1.0f, 1.0f;
corners[2] << 1.0f, -1.0f, 1.0f;
corners[3] << -1.0f, -1.0f, 1.0f;
corners[4] << 1.0f, 1.0f, -1.0f;
corners[5] << -1.0f, 1.0f, -1.0f;
corners[6] << 1.0f, -1.0f, -1.0f;
corners[7] << -1.0f, -1.0f, -1.0f;
// Then transform back into worldspace.
for ( Vector3f& c : corners ) {
c = mult( mvp.inverse(), c );
}
Looks respectable enough, but if I put in my MVP (column-major):
-1.68199 0.0 1.68199 0.0
-0.720796 3.00332 -0.720796 0.0
-0.671735 -0.322433 -0.671735 35.8534
-0.669589 -0.321403 -0.669589 37.3363
This is a camera with near clip at 0.8, far at 500.0, FOV at 35.0°, viewport size of 800x600, positioned at [25.0, 12.0, 25.0] looking at the origin (y is the up axis). I get these corners:
-0.988515, 0.00116634, -0.393982
-0.393982, 0.00116634, -0.988515
-0.845201, -0.595974, -0.250669
-0.250669, -0.595974, -0.845201
30.2115, 14.9772, 30.8061
30.8061, 14.9772, 30.2115
30.3548, 14.38, 30.9494
30.9494, 14.38, 30.3548
Don't stare at the numbers too long (you'll go mad), just note that there are no corners hundreds of units from each other which is what you would expect with a far plane of 500.0. What am I not understanding about this procedure?
Without looking at your numbers that heavily, I just make a wild guess:
I hope you didn't forget to divide your transformed corners by their 4th component after extending them to 4D homogenous coordinates (by adding a 1) and multplying them by the inverse MVP.
This is neccessary because the MVP matrix (actually the projection part) and its inverse is not an affine transformation and thus results in a w-value other than 1 when applied to a homogenous vector. This is the same when you (or actually your shader) apply the usual forward MVP multiplication when transforming vectors, just that the fixed function graphics hardware does this division by w for you (so called perspective division, because this is what actually causes farther away objects to be smaller on screen).