I've been in game development for a while and since then this is the question that I couldn't find an answer to.
So, seriously how does this magic works on its very bases? For example, when I'm using Unreal Engine 4, I can simply call LineTraceByChannel, pick a start and end point, and then the engine will trace a line and return me the hit result. I can also visualize it. But my brain thinks like:
"Okay there is no such a thing as shooting a line segment, like a bullet, this is a simpler way to visualize it. There must be huge math behind it. How does it detect the intersecting actors? Does it check for every available geometry or only the ones that are in the radius of endpoint - start point? Then how does it detects that these actors are in that radius? How come it can be not 'that' much expensive?"
I would be really enlightened if someone can explain what the heck is behind the line tracing hit tests... Thank you for your time...
Nice question. To dense to obtain a short answer of the kind "raycasting works just like this" and that's it. The big deal is how the raycasting and the collision point obtained is optimized and all the math and software tricks behind it for it to be that cheap. However, that's too deep and complex stuff to be explained in an answer I believe. That does not mean that I know the answer.
An approximate unoptimized approach can be to check with sphere-line intersection the entities of the scene. The ray is just a line, so two points. Then, once you got the entities interesected, you get the lowest z coordinate one (closest to the camera where the ray is thrown from), and for all the polygons of that entity, you check for the plane-line for all the planes of the 3d entity model. Again the big deal is how this is optimized to be so computationally cheap and be used for example in an Update()
The same question arises with collisions. In 2D it's not that easy but you can build yourself, for example, a simple collision system, to check if two polygons collide, just checking for example for all of the points of one of the polygons if any is inside the other. To optimize that regarding efficiency and/or precision convex hull and bounding volumes (box, sphere, capsule, cylinder, etc) obtention techniques are used. That goes out of control when you go to 3D.
As to explain even the stuff of the bounding volume techniques used for collision detection and optimization is too large and deep to be explained on an answer to explain collisions in game engines, something similar should be happening to explain how the raycast stuff works in detail I believe.
Anyhow I will be also glad to read any comments about how raycast works in-game engines in detail, maybe some explanation at a high level, beyond the simple model example I explained, which I don't know if it even used actually, its just a computational geometry approach to achieve the intersection point.