Weird result with MSAA
MSAA using OpenGL.
I just drew a white sphere using 'glutSolidSphere' and filled black where 'dot(Normal, CameraVec) < threshold' for silhouette.
And I found a weird result at the outline of the inner white circle. It looks like MSAA not worked.
By the way, it worked well at the outline(outmost) of the black circle.
If I increase the number of samples, it works well even at the outline of the inner white circle.
I think it should work well independent of the number of samples, because resolving samples occurs after fragment shader.
Is this the right result? If yes, why?
Below is the result of 4 samples(left) and 32 samples(right).
MSAA only helps to smooth polygon edges and intersections. It does nothing to smoothen sharp transitions created by your shader code.
The main idea behind MSAA is that the fragment shader is still executed only once per fragment. Each fragment has multiple samples, and coverage is determined by sample. This means that some samples of the fragment can be inside the rendered polygon, and some outside. The fragment shader output is then written to only the covered samples. But all the covered samples within the fragment get the same value.
The depth buffer also has per-sample resolution, meaning that intersections between polygons also profit from the smoothing produced by MSAA.
Once you are aware how MSAA works, it makes sense that it does nothing for sharp transitions in the interior of polygons, which can be the result of logic applied in the shader. To achieve smoothing in this case, you would have to evaluate the fragment shader per sample, which does not happen with MSAA.
MSAA is attractive because it does perform sufficient anti-aliasing for many use cases, with relatively minimal overhead. But as you noticed, it's obviously not sufficient for all cases.
What you can do about this goes beyond the scope of an answer here. There are two main directions:
- You can avoid generating sharp transitions in your shader code. If you use standard texturing, using mipmaps can help. For procedural transitions, you can smooth them out in your code, possibly based on gradient values.
- You can use a different anti-aliasing method. There are too many to mention here. It's easy to get perfect anti-aliasing with super-sampling, but it's very expensive. Most methods try to achieve a compromise in getting better results than plain MSAA, while not adding too much overhead.
I'm somewhat puzzled by the fact that you get some smoothing on the inside edge with 32x MSAA. I don't think that's expected. I wonder if there's something going on during the downsampling that produces some form of smoothing.