I'm trying to implement some simple dithering in my Metal fragment shader, to get rid of banding across gradients. It is not working, and I am wondering if it is just a bug or if the color values passed to the shader are already quantized (if that's the right word) to 8 bits. In other words, the fragment shader deals with floating point values, but are those already at the discrete levels imposed by 8 bit rgb space?
Here is my shader and the matrix I'm using for dithering. I based this on these two articles/posts:
OpenGL gradient "banding" artifacts
http://www.anisopteragames.com/how-to-fix-color-banding-with-dithering/
var dither_pattern:[UInt8] =
[0, 32, 8, 40, 2, 34, 10, 42, /* 8x8 Bayer ordered dithering */
48, 16, 56, 24, 50, 18, 58, 26, /* pattern. Each input pixel */
12, 44, 4, 36, 14, 46, 6, 38, /* is scaled to the 0..63 range */
60, 28, 52, 20, 62, 30, 54, 22, /* before looking in this table */
3, 35, 11, 43, 1, 33, 9, 41, /* to determine the action. */
51, 19, 59, 27, 49, 17, 57, 25,
15, 47, 7, 39, 13, 45, 5, 37,
63, 31, 55, 23, 61, 29, 53, 21]
// this array is passed to the frag shader via a MTLBuffer
fragment float4 window_gradient_fragment(WindowGradientVertexOut interpolated [[stage_in]],
const device unsigned char* pattern [[ buffer(0) ]]) {
int x = (int)interpolated.position.x % 8;
int y = (int)interpolated.position.y % 8;
int val = pattern[x+y*8];
float bayer = 255.0 * (float)val / 64.0;
const float rgbByteMax = 255.0;
float4 rgba = rgbByteMax*interpolated.color;
float4 head = floor(rgba);
float4 tail = rgba-head;
float4 color = head+step(bayer,tail);
return color/255.0;
}
I tested this by changing the matrix to a series of alternating 0,63 pairs, and up close it did produce faint vertical stripes every other pixel. But the overall banding stays the same, from what I can tell. This has me thinking that the banding is already "built-in" by the time it reaches the fragment shader. So applying the dither to it wouldn't really help, because the damage has already been done. I hope that's not true though ... it was my naive assumption that because the colors are floats, they would have full floating point precision and the conversion to 8 bit would happen after the frag shader.
Your logic seems wrong. tail is in [0.0,1.0) by construction. bayer is in [0.0,255.0), more or less. step() is comparing the two. For all values of val >= 1, bayer is >= 3.98, and thus greater than tail. So, it's not very "dithery"; only a val of 0 rounds the color down, all others round up.
I think you just want to not multiply by 255.0 when calculating bayer.