Search code examples
c++openglglm-mathglew

Opengl:Unexpected behaviour when trying to draw a line to visualise mouse Ray cast

I was experimenting with ray casting to implement mouse picking in my application.
What I wanted to do is literally draw the line that is cast from the screen when the mouse is clicked.
Here's what i did:

float xNormalised = ((float)200/w.getWidth())*2 -1;
float yNormalised = -(((float)300/w.getHeight())*2 -1);

glm::vec4 nearP = glm::vec4(xNormalised,yNormalised,-1,1);
glm::vec4 farP = glm::vec4(xNormalised,yNormalised,1,1);

For simplicity i hard code the mouse coordinates from which the ray should be cast, i map them to normalised device coordinates and then i create two vectors representing two points: the point nearest to the screen and the point farthest on the same line. 

worldNear = getInversedPoint(nearP,proj, view);
worldFar = getInversedPoint(farP,proj, view);

Then i call this function on each of the points to basically reverse the pipeline to get their version in the world space, here's what it does:

glm::vec4 getInversedPoint(glm::vec4 point,glm::mat4& proj,glm::mat4 view){

    //glm::mat4 inv = inverse(proj*view);
    point = point*inverse(proj);
    auto v = vec4(point.x/point.w,point.y/point.w,point.z/point.w,1);
    //auto v = vec4(point.x/point.w,point.y/point.w,point.z/point.w,point.w);
    point = v*inverse(view);
    return vec4(point.x,point.y,point.z,point.w);
}

After this i should have gotten the two points in world space if i didn't get it wrong, so i put these two points in a buffer and call gldraw to draw the line that connects them.

What i'm expecting is basically a red dot in the coordinates that i put, because it is a straight line from the click point to the point in front of it... but i get a strange line more or less in the center of the screen, which is not where i specified..
What am i doing wrong?

enter image description here

You can see the red line in the center, near the sideways triangle.
My vertex shader: 

#version 330 core

layout(location = 1)in vec4 pos;
uniform mat4 projectMat;
uniform mat4 viewMat;

void main(){
    gl_Position = pos;
}

I also tried to multiply the points by the two matrices as well but it only slightly change.

Solution

  • You can use glm::unproject to get you far and near points :

    glm::vec3 near = glm::unProject(glm::vec3(mouseX, winSize.y - mouseY, 0), // the screen-space coordinate
                                camera.view,
                                camera.projection,
                                glm::vec4(0, 0, winSize)); // your viewport

glm::vec3 far = glm::unProject(glm::vec3(mouseX, winSize.y - mouseY, 0), // the screen-space coordinate
                               camera.view,
                               camera.projection,
                               glm::vec4(0, 0, winSize)); // your viewport

    You could also get the point under the mouse directly by reading the framebuffer :

    /** read depth buffer and get the point under the cursor **/
        float depth;
        glReadPixels(mouseX, winSize.y - mouseY, 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &depth);
        if (depth < 1.0)
            point_on_plane = glm::unProject(glm::vec3(mouseX, winSize.y - mouseY, depth), camera.view,
                                            camera.projection,
                                            glm::vec4(0, 0, winSize));