I have a vertex shader that essentially acts as a height map, the problem is that sometimes it will write nothing to the buffer when the output of a function is undefined. (specifically in a divide by 0 case or in a sqrt(-) case) This results in jagged edges as some of the triangles aren't rendered. Is there a way for me to check for a null or undefined value in glsl?
EDIT: I just need something that could do: f(x) == undefined or f(x) == null, where f(x) is a function outputting a float.
Floating-point division by zero does not yield "undefined". It yields positive or negative infinity (INF), and you can detect an INF value with the isinf() function.
But for sqrt, there's nothing you can do. "Undefined" is not a value; it means "anything could happen".
It is best to sanitize inputs before doing the computation, not afterwards.