Alright, I don't know if you can quite consider it ray casting, but I am basically trying to find which tile on my 2d grid is being hovered over so I can place an object there.
The solution should only pick up the grid and not the buildings on the grid. Building highlighting will be based on if any of the tiles the building occupies is being hovered over.
A great example of what I am trying to achieve is a Factorio like building system.
Edit: The grid is a 2d array which contains all the tile info. The tiles in world are each 2 triangles (Made from a vertex array and index array). The camera is a perspective camera (If Factorio uses Orthographic, I could switch to it if it made things simpler).
Edit 2: The array contains a class called TileInformation which has a couple of things related to what the tile contains and such. The tiles are 1x1 in size and the array is 256x256. (There will be multiple grid pieces which can be individually updated.) All tiles are in a grid with positions represented as int's (Coordinate system is positive and negative.)
Your question is still not clear enough, but I'll make a few assumptions and hope that they meet your situation.
First, we need a picking ray that represents the 2D mouse position in 3D space. For this, we first transform the mouse position from pixel coordinates into normalized device coordinates between -1 and 1:
mouseNDCX = 2 * mousePixelX / viewportWidth - 1
mouseNDCY = 1 - 2 * mousePixelY / viewportHeight
Then, we find the ray. For this, we need the view-projection matrix of your camera. Then:
mouseNear = inverse(viewProjection) * (mouseNDCX, mouseNDCY, 0, 1)
mouseFar = inverse(viewProjection) * (mouseNDCX, mouseNDCY, 1, 1)
This will give you two positions on the picking ray. One on the z-near plane and one on the z-far plane. Do not forget to do the perspective divide:
mouseNear = (1 / mouseNear.w) * mouseNear
mouseFar = (1 / mouseFar.w) * mouseFar
The next step is calculating the intersection point with the grid. This is where your question is heavily underspecified. I assume that the grid lies on a plane that is parallel to one of the principal planes. In the following, this will be the xy-plane. If you have a different plane, you need to replace the corresponding components. I will further assume that the grid is located at height z.
Now, we want to find the intersection, i.e.:
((1 - t) * mouseNear + t * mouseFar).z = z
<=> mouseNear.z + t * (mouseFar.z - mouseNear.z) = z
<=> t = (z - mouseNear.z) / (mouseFar.z - mouseNear.z)
This lets us calculate the intersection point:
intersection = (1 - t) * mouseNear + t * mouseFar
This intersection point lies on the plane of the grid. Finally, we need to find the tile index. For this, we only need to consider the x- and y-components. Assuming that your grid origin is at o and that the tiles have size s, then the tile indices are:
i = floor((intersection.x - o.x) / s)
j = floor((intersection.y - o.y) / s)