Device Load errors when rendering an ARMeshGeometry using Metal

Question

I'm using ARKit with scene reconstruction and need to rendered the captured scene geometry in metal. I can access this geometry through the ARMeshAnchor.geometry, which is a ARMeshGeometry. However when I try rendering it using my custom metal rendering pipeline, nothing renders and I get a bunch of errors like this:

Invalid device load executing vertex function "myVertex" encoder: "0", draw: 3, at offset 4688

Here's a highly simplified version of my code that I've been using for debugging:

struct InOut {
    float4 position [[position]];
};

vertex InOut myVertex(
    uint vid [[vertex_id]],
    const constant float3* vertexArray [[buffer(0)]])
{
    TouchInOut out;

    const float3 in = vertexArray[vid];
    out.position = float4(in.position, 1);
}

fragment float4 myFragment(TouchInOut in [[stage_in]]){
    return float4(1, 0, 0, 1);
}

// Setup MTLRenderPipelineDescriptor
let pipelineDescriptor = MTLRenderPipelineDescriptor()
pipelineDescriptor.colorAttachments[0].pixelFormat = .rgba8Unorm
pipelineDescriptor.sampleCount = 1

pipelineDescriptor.vertexFunction = defaultLibrary.makeFunction(name: "myVertex")
pipelineDescriptor.fragmentFunction = defaultLibrary.makeFunction(name: "myFragment")

let vertexDescriptor = MTLVertexDescriptor()
vertexDescriptor.attributes[0].format = .float3
vertexDescriptor.attributes[0].offset = 0
vertexDescriptor.attributes[0].bufferIndex = 0
        
vertexDescriptor.layouts[0].stride = MemoryLayout<SIMD3<Float>>.stride

pipelineDescriptor.vertexDescriptor = vertexDescriptor

func render(arMesh: ARMeshAnchor) -> void {

   // snip... — Setting up command buffers

    let renderEncoder = commandBuffer.makeRenderCommandEncoder(descriptor: renderPassDescriptor)!
    renderEncoder.setViewport(MTLViewport(originX: 0, originY: 0, width: 512, height: 512, znear: 0, zfar: 1))
 
        
    renderEncoder.setRenderPipelineState(renderPipelineState)
        
    let vertices = arMesh.geometry.vertices
    let faces = arMesh.geometry.faces
            
    renderEncoder.setVertexBuffer(vertices.buffer, offset: 0, index: 0)
            
    renderEncoder.drawIndexedPrimitives(type: .triangle, indexCount: faces.count * 3, indexType: .uint32, indexBuffer: buffer, indexBufferOffset: 0)
    
    renderEncoder.endEncoding()

   // snip... — Clean up
}

I can't figure out why this code causes the metal exception. It stops throwing if I cap vid in the shader to around 100, but it still doesn't draw anything properly

What's going on here? Why does my code produce an error and how can I fix it?

Answer 1

The problem here is the alignment/packing of the vertex data.

Each vertex in ARMeshGeometry.vertices consists of 3 float components, for a total size of 12 bytes. The code above assumes that this means the data is a float3 / SIMD3<Float>, however the vertices from ARMeshGeometry are actually tightly packed. So while SIMD3<Float> has a stride of 16, the actual vertex data has a stride of 12.

The larger size of float3 (16) vs the actual size of elements in the vertices buffer (12) results in metal trying to access data off the end of the vertices buffer, producing the error.

There are two important fixes here:

1. Make sure the MTLVertexDescriptor has the correct stride:

let exampleMeshGeometry: ARMeshGeometry = ...

vertexDescriptor.layouts[0].stride = exampleMeshGeometry.vertices.stride

2. In the shader, use packed_float3 instead of float3

vertex InOut myVertex(
    uint vid [[vertex_id]],
    const constant packed_float3* vertexArray [[buffer(0)]])
{
    ...
}

After fixing these issues, you should be able to properly transfer ARMeshGeometry buffers to your metal shader