Convert projection to orthogonal raycast

Question

I have a completely implemented, working engine in OpenGL that supports a projection camera with raycasting. Recently, I implemented an orthogonal camera type, and visually, it's working just fine. For reference, here's how I compute the orthographic matrix:

double l = -viewportSize.x / 2 * zoom_;
double r = -l;
double t = -viewportSize.y / 2 * zoom_;
double b = -t;
double n = getNear();
double f = getFar();

m = Matrix4x4(
    2 / (r - l),
    0,
    0,
    -(r + l) / (r - l),
    0,
    2 / (t - b),
    0,
    -(t + b) / (t - b),
    0,
    0,
    -2 / (f - n),
    -(f + n) / (f - n),
    0,
    0,
    0,
    1);

However, my issue now is that raycasting does not work with the orthogonal camera. The issue seems to be that the raycasting engine was coded with projection-type cameras in mind, therefore when using the orthographic matrix instead it stops functioning. For reference, here's a high-level description of how the raycasting is implemented:

1. Get the world-space origin vector

   • Get normalized screen coordinate from input screen coordinates
   • Build mouseVector = (normScreenCoords.x, normScreenCoords.y, 0 if "near" or 1 if "far"
   • Build view-projection matrix (get view and projection matrices from Camera and multiply them)
   • Multiply the mouseVector by the inverse of the view-projection matrix.

2. Get the world-space forward vector

   • Get mouse world coordinates (far) and subtract them from mouse world coordinates (near)

3. Send the world-space origin and world-space forward vectors to the raycasting engine, which handles the logic of comparing these vectors to all the visible objects in the scene efficiently by using bounding boxes.

How do I modify this algorithm to work with orthographic cameras?

Answer 1

Your steps are fine and should work as expected with an orthographic camera. There may be a problem with the way you are calculating the origin and direction.

1.) Get the origin vector. First calculate the mouse position in world-space units, ie float rayX = (mouseX - halfResolution) / viewport.width * (r - l) or similar. It should be offset so the center of the screen is (0, 0), and the extreme values the mouse can reach translate to the edges of the viewport l, r, t, b. Then start with the camera position in world space and add two vectors rayX * camera.local.right and rayY * camera.local.up, where right and up are unit vectors in the camera's local co-ordinate system.

2.) The world space forward vector is always the camera forward vector for any mouse position.

3.) This should work without modification as long as you have the correct vectors for 1 and 2.