Calculating homography of multiple markers on a single plane

I've got an algorithm that calculates the homography of a number of markers in a single image, using OpenCV's findHomography. This works fine, but due to the size of the markers/resolution of the images, the rotation calculated is often quite inaccurate (mainly around the x and z axes of the marker, assuming y is up). I'd love to find a way to improve this.

I have a hypothesis that the fact that I know all the markers are on the same plane (they're on a table) should help improve the homography - I then just need to work out the rotation of the marker around a single axis - but I don't know enough to put this in to practice. Can anyone give me any guidance on this?

This is an example of the situation I'm trying to deal with (the images on the markers are different in reality).

enter image description here

Solution

I like the idea of getting higher precision for the estimated homography by using multiple marker images spread across the scene.

This approach can be realized using Nonlinear Optimization. But it’s a little tricky.

Assumptions, Model, Data

Let’s take your example of 3 printed marker images placed across a table and assume all your assumptions hold.

And now let’s assume that for each of the 3 printed marker images you’ve found N point correspondences.

Point correspondences here means that for a printed marker image you can associate N image pixel positions with corresponding N real world positions (x,y) in centimeters (or some other unit) with respect to a known real world "printed marker coordinate system" (for example, with coordinate (0,0) located at the center of a marker image).

The problem is that the "printed marker coordinate system" is only known locally for each printed marker image. What we need is to bring those coordinate systems together and find a way to express "printed marker coordinate system" positions in a single, global coordinate system. So we need to figure out a relation between them.

Relation between each marker’s homographies

We want to compute a homography which uses all the correspondence data from different printed marker images. Therefore, it is important to understand how they relate to each. Specifically, we want to find a common coordinate system for all marker images in order to formulate an optimization problem.

Now, based on the point correspondences, and using conventional methods, compute 3 homographies:

H1 maps "printed marker coordinate system" positions of marker image #1 to image pixel coordinates
H2 maps "printed marker coordinate system" positions of marker image #2 to image pixel coordinates
H3 maps "printed marker coordinate system" positions of marker image #3 to image pixel coordinates

Let’s assume shortly that no measurement and computation errors are present (just hypothetically). Then, the homographies differ only in a rotation $\varphi$ and translation $(t_x,t_y)$ of the respective "printed marker coordinate systems".

Then, the following equations hold:

$H_1 &= H_2 \cdot A(\varphi^{(2)}, t_{x}^{(2)}, t_{y}^{(2)})$

$H_1 &= H_3 \cdot A(\varphi^{(3)}, t_{x}^{(3)}, t_{y}^{(3)})$

with the rotation and translation defined as follows:

$A(\varphi, t_x, t_y) := \begin{pmatrix}\cos(\varphi)&-\sin(\varphi)&t_x\\\sin(\varphi)&\cos(\varphi)&t_y\\0&0&1\end{pmatrix}$

Putting everything together

That means that H1 equals H2 "up to" a rotation $\varphi$ and translation $(t_x^{(2)},t_y^{(2)})$ and H1 equals H3 "up to" a rotation $\varphi$ and translation $(t_x^{(3)},t_y^{(3)})$ .

This gives us a way for transforming positions of marker image #2 into the coordinate system of marker image #1 (case for positions of marker image #3 can be handled analogous):

$\begin{pmatrix}x^{(1)}\\y^{(1)} \end{pmatrix}:=\begin{pmatrix}\cos(\varphi^{(2)})&-\sin(\varphi^{(2)})\\\sin(\varphi^{(2)})&\cos(\varphi^{(2)})\\ \end{pmatrix}\cdot\begin{pmatrix}x^{(2)}\\y^{(2)} \end{pmatrix} + \begin{pmatrix}t_x^{(2)}\\t_y^{(2)} \end{pmatrix}$

Now, in a real world scenario, these equations will hold only approximately. But nevertheless they give a motivation for estimating the rotation and translation between any pair of printed marker images:

$H_2^{-1}H_1\approx \begin{pmatrix}\cos(\varphi^{(2)})&-\sin(\varphi^{(2)})&t_x^{(2)}\\\sin(\varphi^{(2)})&\cos(\varphi^{(2)})&t_y^{(2)}\\0&0&1\end{pmatrix}$

$H_3^{-1}H_1\approx \begin{pmatrix}\cos(\varphi^{(3)})&-\sin(\varphi^{(3)})&t_x^{(3)}\\\sin(\varphi^{(3)})&\cos(\varphi^{(3)})&t_y^{(3)}\\0&0&1\end{pmatrix}$

Now, using atan2, we can find estimates of six auxiliary variables $(\varphi^{(2)},t_x^{(2)},t_y^{(2)},\varphi^{(3)},t_x^{(3)},t_y^{(3)})$ which describe how the printed marker images #2 and #3 are located relative to #1.

That’s all we need for preparation to have start values and a model to formulate a nonlinear optimization problem.

Nonlinear optimization problem

The final goal is to find 8 coefficients of a homography matrix H which uses all point correspondences from all printed marker images at the same time:

$H=\begin{pmatrix}h_{11}&h_{12}&h_{23}\\h_{21}&h_{22}&h_{23}\\h_{31}&h_{32}&1 \end{pmatrix}$

Using the above model, we can define a system of nonlinear equations. It will have 8 + 6 unknowns:

8 homography coefficients
6 auxiliary variables $(\varphi^{(2)},t_x^{(2)},t_y^{(2)},\varphi^{(3)},t_x^{(3)},t_y^{(3)})$ which describe how the correspondences from marker image #2 and #3 will be mapped to the marker image #1 coordinate system

Using the N point correspondences of 3 marker images, we get N*3*2 = N*6 equations. That means, we need N>=3 point correspondences in order to have more equations than unknowns. Interestingly, the minimum N=3 correspondences per marker image in case of the multiple marker image scenario is less than the minimum N=4 in the case of a single marker image scenario.

I’ll not add the formulation of the nonlinear optimization problem in more detail at this point as the general idea of how to approach the problem was already explained.

For solving the nonlinear optimization problem, Newton’s method can be used, for example.

Remark

All matrices in the above equations need to be normalized like follows:

$H=\begin{pmatrix}\ast&\ast&\ast\\\ast&\ast&\ast\\\ast&\ast&1 \end{pmatrix}$