Projecting line from camera

Question

I'm trying to convert a viewport click onto a world position for an object.

It would be quite simple if all I wanted was to draw a point exactly where the user clicks in the canvas:

void Canvas::getClickPosition(int x, int y, Vector3d(&out)[2]) const
{
    Vector4d point4d[2];
    Vector2d point2d(x, y);
    int w = canvas.width();
    int h = canvas.height();
    Matrix4d model = m_world * m_camera;

    for (int i = 0; i < 2; ++i) {
        Vector4d sw(point2d.x() / (0.5 * w) - 1,
            point2d.y() / (0.5* h) - 1, i * 1, 1);

        point4d[i] = (m_proj * model).inverse() * sw;
        out[i] = point4d.block<1, 3>(0, 0);
    }
}

The expected behavior is achieved with this simple code.

The problem arises when I try to actually make a line that will look like a one pixel when the user first clicks it. Until the camera is rotated in any direction it should look like it was perfectly shot from the camera and that it has whatever length (doesn't matter).

I tried the obvious:

Vector4d sw(point2d.x() / (0.5 * w) - 1,
    point2d.y() / (0.5* h) - 1, 1, 1); // Z is now 1 instead of 0.

The result is, as most of you guys should expect, a line that pursues the vanishing point, at the center of the screen. Therefore, the farther I click from the center, the more the line is twitched from it's expected direction.

What can I do to have a line show as a dot from the click point of view, no matter where at the screen?

EDIT: for better clarity, I'm trying to draw the lines like this:

    glBegin(GL_LINES);
    line.p1 = m_proj * (m_world * m_camera) * line.p1;
    line.p2 = m_proj * (m_world * m_camera) * line.p2;

    glVertex3f(line.p1.x(), line.p1.y(), line.p1.z());
    glVertex3f(line.p2.x(), line.p2.y(), line.p2.z());
    glEnd();

Answer 1

Your initial attempt is actually very close. The only thing you are missing is the perspective divide:

out[i] = point4d.block<1, 3>(0, 0) / point4d.w();

Depending on your projection matrix, you might also need to specify a z-value of -1 for the near plane instead of 0.

And yes, your order of matrices projection * model * view seems strange. But as long as you keep the same order in both procedures, you should get a consistent result.

Make sure that the y-axis of your window coordinate system is pointing upwards. Otherwise, you will get a result that is reflected at the horizontal midline.