Better control over Tessellation in OpenGL?

Question

I spent the day working on an OpenGL application that will tessellate a mesh and apply a lens distortion. The goal is to be able to render wide angle shots for a variety of different lenses. So far I've got the shaders properly applying the distortion but I've been having issues controlling the tessellation the way I want to. Right now my Tessellation Control Shader just breaks a single triangle into a set number of smaller triangles, then I apply the lens distortion in in the Tessellation Evaluation Shader.

The problem I'm having with this approach is that when I have really large triangles in the scene, they tend to need more warping. This means they need to be tessellated more in order to ensure good looking results. Unfortunately, I can't compute the size of a triangle (in screen space) in the Vertex Shader or the Tessellation Control Shader, but I need to define the tessellation amount in the Tessellation Control shader.

My question is then, is there some way to get a hold of the entire primitive in OpenGL's programmable pipeline, compute some metrics about it, then use that information to control tessellation?

Here's some example images of the problem for clarity...

Figure 1 (Above): Each Red or Green Square was originally 2 triangles, this example looks good because the triangles were small.

Figure 2 (Above): Each Red or Green Region was originally 2 triangles, this example looks bad because the triangles were small.

Figure 3 (Above): Another example with small triangles but with a much, much larger grid. Notice how much things curve on the edges. Still looks good with tessellation level of 4.

Figure 4 (Above): Another example with large triangles, only showing center 4 columns because the image is unintelligible if more columns are present. This shows how very large triangles don't get tessellated well. If I set the tessellation really really high then this comes out nice. But then I'm performing a crazy amount of tessellation on smaller triangles too.

Answer 1

In a Tessellation Control Shader (TCS) you have read access to every vertex in the input patch primitive. While that sounds nice on paper, if you are trying to compute the maximum edge length of a patch, it would actually mean iterating over every vertex in the patch on every TCS invocation and that's not particularly efficient.

Instead, it may be more practical to pre-compute the patch's center in object-space and determine the radius of a sphere that tightly bounds the patch. Store this bounding information as an extra vec4 attribute per-vertex, packed as shown below.

Pseudo-code for a TCS that computes the longest length of the patch in NDC-space

#version 420

uniform mat4 model_view_proj;

in vec4 bounding_sphere []; // xyz = center (object-space), w = radius

void main (void)
{
  vec4  center = vec4 (bounding_sphere [0].xyz, 1.0f);
  float radius =       bounding_sphere [0].w;

  // Transform object-space X extremes into clip-space
  vec4 min_0 = model_view_proj * (center - vec4 (radius, 0.0f, 0.0f, 0.0f));
  vec4 max_0 = model_view_proj * (center + vec4 (radius, 0.0f, 0.0f, 0.0f));

  // Transform object-space Y extremes into clip-space
  vec4 min_1 = model_view_proj * (center - vec4 (0.0f, radius, 0.0f, 0.0f));
  vec4 max_1 = model_view_proj * (center + vec4 (0.0f, radius, 0.0f, 0.0f));

  // Transform object-space Z extremes into clip-space
  vec4 min_2 = model_view_proj * (center - vec4 (0.0f, 0.0f, radius, 0.0f));
  vec4 max_2 = model_view_proj * (center + vec4 (0.0f, 0.0f, radius, 0.0f));

  // Transform from clip-space to NDC
  min_0 /= min_0.w; max_0 /= max_0.w;
  min_1 /= min_1.w; max_1 /= max_1.w;
  min_2 /= min_2.w; max_2 /= max_2.w;

  // Calculate the distance (ignore depth) covered by all three pairs of extremes
  float dist_0 = distance (min_0.xy, max_0.xy);
  float dist_1 = distance (min_1.xy, max_1.xy);
  float dist_2 = distance (min_2.xy, max_2.xy);

  // A max_dist >= 2.0 indicates the patch spans the entire screen in one direction
  float max_dist = max (dist_0, max (dist_1, dist_2));

  // ...
}

If you run your 4^th diagram through this TCS, you should come up with a value for max_dist very nearly 2.0, which means you need as much subdivision as possible. Meanwhile, many of the patches on the periphery of the sphere in the 3^rd diagram will be close to 0.0; they don't need much subdivision.

This does not properly deal with situations where part of the patch is offscreen. You would need to clamp the NDC extremes to [-1.0,1.0] to properly handle those situations. Seemed like more trouble than it was worth.