I'm using
worldview_inverse * (projection_inverse * vector)
to transform screen space coordinates into world space coordinates. I assumed that
(x,y,1,1)
would transform to a point on the far plane, while
(x,y,-1,1)
transforms to a point on the near plane, and connecting the line I can query all objects in the view frustum that intersect the line. After the transformation I divide the resulting points by their respective .w component. This works for the far-plane, but the point on the near plane somehow gets transformed to the world space origin.
I think this has to do with the w components of 1 I'm feeding into the inverse projection, because usually it is 1 before projection, not after, and I'm doing the reverse projection. What am I doing wrong?
I know this is only a workaround, but you can deduce the near plane point by only using the far point and the viewing position.
near_point = view_position
+ (far_point - view_position) * (near_distance / far_distance)
As for you real problem. First, don't forget to divide by W! Also, depending on your projection matrix, have you tried (x,y,0,1) as opposed to z=-1.
near_point = worldview_inverse * (projection_inverse * vector)
near_point /= near_point.W