How to find Camera matrix for Augmented Reality?

Question

I want to augment a virtual object at x,y,z meters wrt camera. OpenCV has camera calibration functions but I don't understand how exactly I can give coordinates in meters

I tried simulating a Camera in Unity but don't get expected result.

I set the projection matrix as follows and create a unit cube at z = 2.415 + 0.5 . Where 2.415 is the distance between the eye and the projection plane (Pinhole camera model) Since the cube's face is at the front clipping plane and it's dimension are unit shouldn't it cover the whole viewport?

    Matrix4x4 m = new Matrix4x4();
    m[0, 0] = 1;
    m[0, 1] = 0;
    m[0, 2] = 0;
    m[0, 3] = 0;

    m[1, 0] = 0;
    m[1, 1] = 1;
    m[1, 2] = 0;
    m[1, 3] = 0;

m[2, 0] = 0;
    m[2, 1] = 0;
    m[2, 2] = -0.01f;
    m[2, 3] = 0;

    m[3, 0] = 0;
    m[3, 1] = 0;
    m[3, 2] = -2.415f;
    m[3, 3] = 0;

Answer 1

I ended up measuring the field of view manually. Once you know FOV you can easily create the projection matrix. No need to worry about units because in the end the projection is of the form ( X*d/Z, Y*d/Z ). Whatever the units of X,Y,Z may be the ratio of X/Z remains the same.