How big should epsilon be when checking if dot product is close to 0?
I am working on raytracing project, and i need to check if the dot product is 0. But that will probably never happen, so I want to take it as 0 if its value is in a small area [-eps, +eps], but I am not sure how big should eps be?
Thanks
Since you describe this as part of a ray-tracing project, your accuracy needed is likely dictated by the "world coordinates" of a scene, or perhaps even the screen coordinates to which those are translated. Either of these would give an acceptable target for absolute error in your calculation.
It might be possible to backtrack from that to the accuracy required in the intermediate calculations you are doing, such as forming an inner product which is supposed "theoretically" to be zero. For example, you might be trying to find a shortest path (reflected light) between two smooth bodies, and the disappearance of the inner product (perpendicularity) gives the location of a point.
In a case like that the inner product may be a quadratic in the unknowns (location of a point) that you seek. It's possible that the unknowns form a "double root" (zero of multiplicity 2), making the location of that root extra sensitive to the computation of the inner product being zero.
For such cases you would want to get roughly twice the number of digits "zero" in the inner product as needed in the accuracy of the location. Essentially the inner product changes very slowly with location in the neighborhood of a double root.
But your application might not be so sensitive; analysis of the algorithm involved is necessary to give you a good answer. As a general rule I do the inner product in double precision to get answers that may be reliable as far as single precision, but this may be too costly if the ray-tracing is to be done in real time.