Search code examples
openglglsldepth-buffercullingocclusion-culling

GLSL shader: occlusion order and culling

I have a GLSL shader that draws a 3D curve given a set of Bezier curves (3d coordinates of points). The drawing itself is done as I want except the occlusion does not work correctly, i.e., under certain viewpoints, the curve that is supposed to be in the very front appears to be still occluded, and reverse: the part of a curve that is supposed to be occluded is still visible.

To illustrate, here are couple examples of screenshots:

  1. Colored curve is closer to the camera, so it is rendered correctly here.

    correct render

  2. Colored curve is supposed to be behind the gray curve, yet it is rendered on top.

    wrong render

I'm new to GLSL and might not know the right term for this kind of effect, but I assume it is occlusion culling (update: it actually indicates the problem with depth buffer, terminology confusion!).

My question is: How do I deal with occlusions when using GLSL shaders?

Do I have to treat them inside the shader program, or somewhere else?

Regarding my code, it's a bit long (plus I use OpenGL wrapper library), but the main steps are:

  1. In the vertex shader, I calculate gl_Position = ModelViewProjectionMatrix * Vertex; and pass further the color info to the geometry shader.
  2. In the geometry shader, I take 4 control points (lines_adjacency) and their corresponding colors and produce a triangle strip that follows a Bezier curve (I use some basic color interpolation between the Bezier segments).
  3. The fragment shader is also simple: gl_FragColor = VertexIn.mColor;.

Regarding the OpenGL settings, I enable GL_DEPTH_TEST, but it does not seem to have anything of what I need. Also if I put any other non-shader geometry on the scene (e.g. quad), the curves are always rendered on the top of it regardless the viewpoint.

Any insights and tips on how to resolve it and why it is happening are appreciated.

Update solution

So, the initial problem, as I learned, was not about finding the culling algorithm, but that I do not handle the calculation of the z-values correctly (see the accepted answer). I also learned that given the right depth buffer set-up, OpenGL handles the occlusions correctly by itself, so I do not need to re-invent the wheel.

I searched through my GLSL program and found that I basically set the z-values as zeros in my geometry shader when translating the vertex coordinates to screen coordinates (vec2( vertex.xy / vertex.w ) * Viewport;). I had fixed it by calculating the z-values (vertex.z/vertex.w) separately and assigned them to the emitted points (gl_Position = vec4( screenCoords[i], zValues[i], 1.0 );). That solved my problem.

Regarding the depth buffer settings, I didn't have to explicitly specify them since the library I use set them up by default correctly as I need.

Solution

  • If you don't use the depth buffer, then the most recently rendered object will be on top always.

    You should enable it with glEnable(GL_DEPTH_TEST), set the function to your liking (glDepthFunc(GL_LEQUAL)), and make sure you clear it every frame with everything else (glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT)).

    Then make sure your vertex shader is properly setting the Z value of the final vertex. It looks like the simplest way for you is to set the "Model" portion of ModelViewProjectionMatrix on the CPU side to have a depth value before it gets passed into the shader.

    As long as you're using an orthographic projection matrix, rendering should not be affected (besides making the draw order correct).