Plane reprojection in post-process using homography matrix

Question

Let's take a proxy problem - You're looking at some 3D scene. Now I replace the scene with its rendering done from your eyes position, so you don't see any difference. Then I replace the rendered image with a wall and projector. What I need is an image, that when projected on the wall will look exactly as if you were looking at the scene.

It looks like this: The left camera is the observer, the right one is projector.

My approach is to render the scene from observers location, then in the post process I sample rendered image to add distortion.

I have some proof of concept code that kind of works up to some offsets that I need to debug, but most of the computations is done in pixel shader so it's not the best solution.

After I did my initial version I read about homography matrices and it seems to be the right tool for my needs. If I understand it correctly I should be able to compute the homography matrix and then only multiply my screen space UV with that to get reprojected UVs.

Ufortunately most of the info about homography I could find relates to the case when I have 2 pictures of some object, pick by hand 4 corresponding point pairs and compute a matrix from that, but I don't have such points. Instead, I know the exact transforms of both views, their vertical and horizontal FOV and plane so I think that's all I need.

The perfect solution would be to have some transform that maps my image-space UVs in [0,1] range to correct UV for texture sampling. Did any of you saw similar solution?

EDIT:

I've made a screenshot from my POC implementation: I'm not sure it's 1:1 correct but it shows what I need. The rendering is distorted, but this distortion should cancel out when I project this image on the wall. Please take a look at the table top - it still look like rendered from the side and not from projector. When I project this image on the wall it should look like rendered from your eyes position.

Answer 1

I've finally found a solution.

In some of the articles about Homography I saw an equation to compute H matrix from known transformations of two viewports but couldn't get it to work, maybe I've misunderstood something or had my math wrong so I decided to try the 4-points approach, but how to get such points automatically?

Given, that I'm going to use this in post-processing stage of my rendering, thus operating mainly in screen-space I decided to map corners of my destination viewport to my source viewport.

From Projector transform I get the forward direction and rotate it by half of vertical and horizontal FOV. Combining positive and negative values I can get 4 vectors that correspond to 4 corners of my viewport, computed in World space.

Using Projector position as a start and 4 computed vectors as directions I can compute 4 intersections with my Plane. Those are World-space points, lying on my Plane that represents corners of what is visible from projector.

Having those 4 points I project them with Observer camera to its Screen-space.

The points I get here are the Observer-screen-space points and given that I've mapped entire viewport of Projector I can use typical [0,0],[1,0],[0,1],[1,1] rectangle as a Projector-screen-space points.

This way I've got four corresponding pairs of points to use with my Homography.

Homography

This was a bit trickier as I still don't fully understand the math here, but I've used this: http://www.cse.psu.edu/~rtc12/CSE486/lecture16.pdf article, on page 29 there's a matrix equation using 8x8 square matrix. The link may disappear but you can find the same equation in a lot of places, what's important is that I've used 8x8 version, but saw notations using 8x9 matrix.

Here's the screen:

In general it's a A*x=b notation where we know A and b and want to compute x.

For computations I've used JacobiSVD class from the Eigen library.

After I get resulting 8 floats I can build 3 row vectors of my matrix I'm looking for. First 3 values form first vector, another 3 values form 2nd vector, then we're left with only 2 values, so we append a 1.0f to get the last vector.

Let's call this matrix H.

Pixel Shader

Having all those computations done per-frame the Pixel shader is really simple - we just need to transform screen-space UV vector with the H matrix (but in Homogeneous coordinates).

We get the u,v values and append 1.0 to get [u,v,1.0] then we multiply it with the H matrix. We've passed this matrix as a row vectors so we can dot product the [u,v,1.0] vector with 3 rows and sum up the results. This way we get a result like [x,y,z] but in fact it means a [x/z,y/z] 2D vector - this is our UV we were looking for. Now I just sample a texture with this UV and I'm done.

Why is it better than Pavel approach?

I don't need a separate geometry rendering, which is especially slow in deferred rendering engines, like Unreal.

I can use few such textures at once, all on different planes. I can e.g. pass few alpha-masked and textured planes and decide which one I hit.

I can reproject a 3D side-by-side rendering to get reprojected side-by-side, by treating each half of a screen separately.