Search code examples
scenekitarkitmetal

ARKit color correction of captured image in low light scenes

I have an ARKit app that does the following:

  • Render the frame's captured image to a texture.
  • Apply this texture to a scenekit object in the AR scene.

I use this to create a virtual object that perfectly blends into the AR scene.

My current approach works great for well-lit scenes, but in dark scenes the texture on the virtual object becomes subtly different than the current scene background. What causes this and is there any way to fix it?

Details

I've created this branch on a demo project which demonstrates the issue.

The project renders a face model that is textured with the frame's currentImage. The result should be that face model effectively becomes invisible, even though it is still being rendered. However in low light situations, you can clearly see where the background image ends and the face model starts.

Here's an overview of the shader I use to capture the texture

// Take in the current captured image. 
fragment float4 fragmentShader(TextureInOut in [[stage_in]],
                               texture2d<float, access::sample> capturedImageTextureY [[texture(0)]],
                               texture2d<float, access::sample> capturedImageTextureCbCr [[texture(1)]])
{
    
    // Sample from the captured image
    const float4 ycbcr = float4(capturedImageTextureY.sample(textureSampler, in.screenSpaceTexCoord).r,
                                capturedImageTextureCbCr.sample(textureSampler, in.screenSpaceTexCoord).rg, 1.0);
    
    
    // Convert to RGB
    const float3 color = (ycbcrToRGBTransform * ycbcr).rgb;
    
    // Gamma correction
    return float4(pow(color, float3(2.2)), 1);
}

This shader takes the frame's captured image and writes out an RGB texture. I do some basic gamma correction so that the resulting texture looks correct.

I store the output in a metal texture (.rgba8Unorm) and then have a SceneKit object that uses that texture for its diffuse component.

Again, this all works great for well-lit scenes. In low light scenes however, the captured texture becomes noticeably different than the background image.

The example image below shows the problem. You can see a clear line (right near the edge of the hair) where the face model end and the background image begins:

enter image description here

This is not visible in well-lit scenes.

The problem is not misalignment, in dark scenes the colors on the virtual face texture seem more saturated and/or less noisy.

What may be causing this and how can I avoid it?

Solution

  • You are using an approximate gamma correction, its not the correct conversion between RGB and sRGB. What you are really trying to do is circumvent SceneKits default pixel format (sRGB). In the fragment shader after you do the YCbCr to RGB conversion you have linear RGB but when you write to a texture from the fragment shader, that value from the shader will be interpreted as sRGB, so an RGB to sRGB conversion will happen i.e (essentially pow 1/2.4, hence why you tried to correct it with an approximate gamma correction), you need to do the inverse to circumvent this, so as if you were going from sRGB to linear. RGB-sRGB conversion (and vice versa) can be confusing as sometimes things are happening underneath the hood that you might not be aware off. So the fix is, instead of your gamma correction do this:

    float3 color2;
color2.x = ( color.r > 0.04045 ) ? pow((color.r + 0.055) / 1.055, 2.4) : color.r / 12.92;
color2.y = ( color.g > 0.04045 ) ? pow((color.g + 0.055) / 1.055, 2.4) : color.g / 12.92;
color2.z = ( color.b > 0.04045 ) ? pow((color.b + 0.055) / 1.055, 2.4) : color.b / 12.92;

return float4(color2, 1.0);

    For more information on this, check out the last page of the current Metal shading language guide (v 2.3).