3d Math : screen space to world space

Question

I've been trying to implement clicking in my webgl app for the last 6 hours and I can't find anything clear enough about this subject.

What I have came up with so far is, in pseudo code:

screenSpace = mousePosition;
normalizedScreenSpace = (screenSpace.x/screen.width, screenSpace.y/screen.height);

camSpace = invertProjectionMatrix * normalizedScreenSpace;

worldSpace = invertViewMatrix * camSpace;

Printing out the worldSpace coordinates, and it doesn't corresponds to other objects in the scene. What am I doing wrong?

Answer 1

The viewProjection matrix brings a vec3 from world space to clip space and so its inverse does the reverse, clip space to world space. Whats missing is the perspective divide that gpu handles for you behind the hood so you have to account for that as well. Add in the screen width and height and you have your screen to world:

screenToWorld: function(invViewProjection, screenWidth, screenHeight){
    // expects this[2] (z value) to be -1 if want position at zNear and +1 at zFar

    var x = 2*this[0]/screenWidth - 1.0;
    var y = 1.0 - (2*this[1]/screenHeight); // note: Y axis oriented top -> down in screen space
    var z = this[2];
    this.setXYZ(x,y,z);
    this.applyMat4(invViewProjection);
    var m = invViewProjection;
    var w = m[3] * x + m[7] * y + m[11] * z + m[15]; // required for perspective divide
    if (w !== 0){
        var invW = 1.0/w;
        this[0] *= invW;
        this[1] *= invW;
        this[2] *= invW;
    }

    return this;
},

And the reverse calculation:

worldToScreen: function(viewProjectionMatrix, screenWidth, screenHeight){
    var m = viewProjectionMatrix;
    var w = m[3] * this[0] + m[7] * this[1] + m[11] * this[2] + m[15]; // required for perspective divide
    this.applyMat4(viewProjectionMatrix);
    if (w!==0){ // do perspective divide and NDC -> screen conversion
        var invW = 1.0/w;
        this[0] = (this[0]*invW + 1) / 2 * screenWidth;
        this[1] = (1-this[1]*invW) / 2 * screenHeight; // screen space Y goes from top to bottom
        this[2] *= invW;
    } 
    return this;
},