Did Apple break texture projection again?

Question

In iOS 8 there was a problem with the division of floats in Metal preventing proper texture projection, which I solved.

Today I discovered that the texture projection on iOS 9 is broken again, although I'm not sure why.

The result of warping a texture on CPU (with OpenCV) and on GPU are not the same. You can see on your iPhone if you run this example app (already includes the fix for iOS 8) from iOS 9.

The expected CPU warp is colored red, while the GPU warp done by Metal is colored green, so where they overlap they are yellow. Ideally you should not see green or red, but only shades of yellow.

Can you:

• confirm the problem exists on your end;
• give any advice on anything that might be wrong?

The shader code is:

struct VertexInOut
{
  float4 position [[ position ]];
  float3 warpedTexCoords;
  float3 originalTexCoords;
};

vertex VertexInOut warpVertex(uint vid [[ vertex_id ]],
                              device float4 *positions [[ buffer(0) ]],
                              device float3 *texCoords [[ buffer(1) ]])
{
  VertexInOut v;
  v.position = positions[vid];

  // example homography
  simd::float3x3 h = {
    {1.03140473, 0.0778113901, 0.000169219566},
    {0.0342947133, 1.06025684, 0.000459250761},
    {-0.0364957005, -38.3375587, 0.818259298}
  };

  v.warpedTexCoords = h * texCoords[vid];
  v.originalTexCoords = texCoords[vid];

  return v;
}

fragment half4 warpFragment(VertexInOut inFrag [[ stage_in ]],
                            texture2d<half, access::sample> original [[ texture(0) ]],
                            texture2d<half, access::sample> cpuWarped [[ texture(1) ]])
{
  constexpr sampler s(coord::pixel, filter::linear, address::clamp_to_zero);
  half4 gpuWarpedPixel = half4(original.sample(s, inFrag.warpedTexCoords.xy * (1.0 / inFrag.warpedTexCoords.z)).r, 0, 0, 255);
  half4 cpuWarpedPixel = half4(0, cpuWarped.sample(s, inFrag.originalTexCoords.xy).r, 0, 255);

  return (gpuWarpedPixel + cpuWarpedPixel) * 0.5;
}

Answer 1

Do not ask me why, but if I multiply the warped coordinates by 1.00005 or any number close to 1.0, it is fixed (apart from very tiny details). See last commit in the example app repo.