How to unproject a point on screen to object space coordinates in vulkan?

I need to be able to unproject a screen pixel into object space using Vulkan, but somewhere my math is going wrong.

Here is the shader as it stands today for reference:

void main()
{
    //the depth of this pixel is between 0 and 1
    vec4 obj_space = vec4( float(gl_FragCoord.x)/ubo.screen_width, float(gl_FragCoord.y)/ubo.screen_height, gl_FragCoord.z, 1.0f);
    //this puts us in normalized device coordinates [-1,1 ] range
    obj_space.xy = ( obj_space.xy * 2.0f ) -1.0f;

    //this two lines will put is in object space coordinates
    //mvp_inverse is derived from this in the c++ side: 
    //glm::inverse(app.three_d_camera->get_projection_matrix() * app.three_d_camera->view_matrix * model);

    obj_space = ubo.mvp_inverse * obj_space;
    obj_space.xyz /= obj_space.w;


    //the resulting position here is wrong
    out_color = obj_space;
}

when I output the position in color, the colors are off. I know I can simply pass in the object space position from the vertex shader to the fragment shader, but I'd like to understand why my math is not working, it will help me understand Vulkan and maybe learn a little math myself.

Thanks!

Solution

I'm not entirely sure what your problem is, but lets go over potential problems.

Remember, vulkan clip space is:

positive y = down,
positive x = right,
positive z = out,
centered at the middle of the screen.

Additionally, despite OpenGL's GLSL docs saying it is centered at the bottom left corner, in vulkan gl_FragCoord is centered at the top left corner.

in this step:

obj_space.xy = ( obj_space.xy * 2.0f ) -1.0f;

obj_space is now:

left x : -1.0
right x : 1.0
top y = -1.0
bottom y = 1.0
out z = 1.0
back z = 0

I'm almost entirely sure you don't mean your object space to have Y be negative at the top. The reasoning for y increasing starting from top to bottom is for images and textures, which on the CPU are ordered the same way, and now are ordered like that in vulkan.

Some other notes:

You claim your inverse is derivied from glm::inverse here:

glm::inverse(app.three_d_camera->get_projection_matrix() * app.three_d_camera->view_matrix * model);

But GLM uses OpenGL notation for matrix dimensions and handedness, and unless you force it to the correct coordinate system, it is going to assume right handed positive Y up, z negative out. You'll need to include the following #defines before it works correctly (or physically change your calculations to accommodate this).

#define GLM_FORCE_DEPTH_ZERO_TO_ONE
#define GLM_FORCE_LEFT_HANDED

Additionally you'll need to modify your matrices to account for the negative Y direction. Here is an example of how I've handled this in the past (modifying the perspective matrix directly):

ubo.model = glm::translate(glm::mat4(1.0f), glm::vec3(pos_x,pos_y,pos_z));
ubo.model *= glm::rotate(glm::mat4(1.0f), time * glm::radians(0.0f), glm::vec3(0.0f, 0.0f, 1.0f));
ubo.view = glm::lookAt(glm::vec3(0.0f, 0.0f, -10.0f), glm::vec3(0.0f, 0.0f, 0.0f), glm::vec3(0.0f, 1.0f, 0.0f));
ubo.proj = glm::perspective(glm::radians(45.0f), swapChainExtent.width / (float) swapChainExtent.height, 0.1f, 100.0f);
ubo.proj[1][1] *= -1; // makes the y axis projected to the same as vulkans