Metal Texture with Overlay comes in Green

Question

I am trying to use Metal to overlay two textures together. The base layer is from the camera a CMSampleBuffer in 420f format and the secondary is a texture generated from a UIImage with no background and has text within it. Due to the nature of the CMSampleBuffer being in 420f the shader is trying to write to the two planes of the underlying camera CMSampleBuffer. The resulting image has a transparent green hue. Since the camera feed is in YCbCr I convert the two textures Y and Cb and Cr to RGBA and and then merge the RGBA of the base texture and overlay texture together. Then converting it back to Y, Cb and Cr and then write to the CMSampleBuffer textures. I have tried a couple of things to get this to work playing with the corresponding RGB values of the overlay to the RGB of the base image, Changing the MTLPixelFormat when creating a texture from an image in CVMetalTextureCacheCreateTextureFromImage. Lastly the odd part is that if I add in an outputTexture to my shader function that expects the RGBA instead of converting and writing the blendedColor to it the images comes out as you would expected.

kernel void combine_chroma_luminance_Overlay(texture2d<float, access::read_write> luminanceTexture [[texture(0)]],
                                     texture2d<float, access::read_write> chromaTexture [[texture(1)]],
                                     texture2d<float, access::read> overlayTexture [[texture(2)]],
                                     texture2d<float, access::write> outTexture [[texture(3)]],
                                     uint2 gid [[thread_position_in_grid]])
{ 
// Get the dimensions
     int2 luminanceSize = int2(luminanceTexture.get_width(), luminanceTexture.get_height());
     int2 chromaSize = int2(chromaTexture.get_width(), chromaTexture.get_height());

// Read the luminance
     float luminance = luminanceTexture.read(gid).r;

// Compute the corresponding position in the chroma texture
     float2 chromaUV = float2(gid) / float2(luminanceSize) * float2(chromaSize);
     uint2 chromaCoord = uint2(chromaUV);

// Read the chroma value
     float2 chroma = chromaTexture.read(chromaCoord).rg;

// Combine the luminance and chroma to form the RGB color
     float4 color;
     color.r = luminance + 1.402 * (chroma.y - 0.5);
     color.g = luminance - 0.344136 * (chroma.x - 0.5) - 0.714136 * (chroma.y - 0.5);
     color.b = luminance + 1.772 * (chroma.x - 0.5);
     color.a = 1.0;

// Read the overlay RGBA
     float4 overlayColor = overlayTexture.read(gid);
     float alpha = overlayColor.a;

// Combine RGBA of baseImage to OverlayImage
     float4 blendedColor;
     blendedColor.r = overlayColor.r * alpha + (1.0 - alpha) * color.r;
     blendedColor.g = overlayColor.g * alpha + (1.0 - alpha) * color.g;
     blendedColor.b = overlayColor.b * alpha + (1.0 - alpha) * color.b;
     blendedColor.a = 1.0;

// Convert blendedColor back to Y CbCr
     float Y = 0.299*blendedColor.r+0.587*blendedColor.g+0.114*blendedColor.b;
     float Cb = 0.564*(blendedColor.b-Y);
     float Cr = 0.713*(blendedColor.r-Y);

// Write to the CMSampleBuffer Textures
     luminanceTexture.write(float4(Y, 0, 0, 1.0), gid);
     chromaTexture.write(float4(Cr, Cb, 0, 0.0), chromaCoord);
     outTexture.write(blendedColor, gid);
}

outTexture of RGBA output: pixelBuffer of written to luminanceTexture and chromaTexture: overlay image on its own:

Answer 1

I had to test before I wrote a full answer, and it seems that what I suggested in the comment was the issue. With a small change the image is no longer greenish.

Change the conversion to:

     float Y = 0.299*blendedColor.r+0.587*blendedColor.g+0.114*blendedColor.b;
     float Cb = 0.564*(blendedColor.b-Y) + 0.5;
     float Cr = 0.713*(blendedColor.r-Y) + 0.5;

This centers the Cb and Cr components around the "neutral" value 0.5.