Search code examples
iosswiftarkitswift5metal

Draw a textured rectangle in Metal - Swift


I am trying to draw a Textured 2D rectangle in Metal using Swift. All of the examples online, including the following, are in Objective-C or older versions of Swift and finding difficult porting to Swift 5:

How to draw a textured rectangle with Metal https://developer.apple.com/documentation/metal/creating_and_sampling_textures

The objective for me is to draw a simple watermark in the corner of my video. I have a Renderer class which is used to capture frames, draw them to my Metal View (this works fine) and then want to draw a logo over the top. I can't use the same approach with the camera frames as they are in a YUV format so are using a special shader. My watermark is just a plain old PNG file.

I am able to draw untextured rectangle, but trying to apply a texture fails - no error, nothing. Just nothing appears on screen. What are the key things I should be checking when Metal returns no errors to debug these types of issues?

Here is some code:

func drawRect(renderEncoder: MTLRenderCommandEncoder, x:Double, y:Double, w:Double, h:Double, pipelineState:MTLRenderPipelineState, viewSize: CGSize)
    {
        //new viewport to size of image we want to display
        var viewPort = MTLViewport(originX: x, originY: y, width: w, height: h, znear: 0.0, zfar: 1.0)
        var viewPortSize = simd_float2(x: Float(w), y: Float(h))
        
        var logoQuadVertices:[AAPLVertex] = [
            //Triangle1: top left, top right, bottom left
            AAPLVertex(position: simd_float2(x: 0, y: 0), textureCoordinate: simd_float2(x: 0.0, y: 0.0)),
            AAPLVertex(position: simd_float2(x: 100, y: 0), textureCoordinate: simd_float2(x: 1.0, y: 0.0)),
            AAPLVertex(position: simd_float2(x: 0, y: 100), textureCoordinate: simd_float2(x: 0.0, y: 1.0)),
            //Triangle2: bottom left, bottom right, top right
            AAPLVertex(position: simd_float2(x: 0, y: 100), textureCoordinate: simd_float2(x: 0.0, y: 1.0)),
            AAPLVertex(position: simd_float2(x: 100, y: 100), textureCoordinate: simd_float2(x: 1.0, y: 1.0)),
            AAPLVertex(position: simd_float2(x: 100, y: 0), textureCoordinate: simd_float2(x: 1.0, y: 0.0))
        ]

        //Set the new viewport
        renderEncoder.setViewport(MTLViewport(originX: x, originY: y, width: w, height: h, znear: 0.0, zfar: 1.0))

        //Debug string
        renderEncoder.pushDebugGroup("DrawRect")

        //Set pipeline state
        renderEncoder.setRenderPipelineState(pipelineState)
    
        //Send vertex data of rectangle to shaders
        renderEncoder.setVertexBytes(logoQuadVertices, length: logoQuadVertices.count*MemoryLayout<AAPLVertex>.stride, index: Int(AAPLVertexInputIndexVertices.rawValue))

        //Send vertex bytes of viewport to shaders
        renderEncoder.setVertexBytes(&viewPortSize, length: MemoryLayout<simd_float2>.stride, index: Int(AAPLVertexInputIndexViewportSize.rawValue))

        // Set the texture
        renderEncoder.setFragmentTexture(texture, index: Int(kTextureIndexPNG.rawValue))

        //Sampler state
        if let samplerState = logoSamplerState {
            renderEncoder.setFragmentSamplerState(samplerState, index: 0)
        }
    
        //Draw the textured rectangle
        renderEncoder.drawPrimitives(type: .triangleStrip, vertexStart: 0, vertexCount: 6)
        
        //Debug
        renderEncoder.popDebugGroup()
    }

...
// Pipeline configuration
let logoPipelineStateDescriptor = MTLRenderPipelineDescriptor()
logoPipelineStateDescriptor.label = "MyLogoPipeline"
logoPipelineStateDescriptor.sampleCount = renderDestination.sampleCount
logoPipelineStateDescriptor.vertexFunction = 
defaultLibrary.makeFunction(name: "vertexShader")!
logoPipelineStateDescriptor.fragmentFunction = 
defaultLibrary.makeFunction(name: "samplingShader")
logoPipelineStateDescriptor.vertexDescriptor = logoPlaneVertexDescriptor
logoPipelineStateDescriptor.colorAttachments[0].pixelFormat = 
renderDestination.colorPixelFormat
logoPipelineStateDescriptor.depthAttachmentPixelFormat = 
renderDestination.depthStencilPixelFormat
logoPipelineStateDescriptor.stencilAttachmentPixelFormat = 
renderDestination.depthStencilPixelFormat
do {
    try logoPipelineState = device.makeRenderPipelineState(descriptor: logoPipelineStateDescriptor)
} catch let error {
    print("Failed to created logo pipeline state, error \(error)")
}
...

Shaders:

typedef struct
{
    float4 position [[position]];
    float2 textureCoordinate;
} RasterizerData;

// Vertex Function
vertex RasterizerData
vertexShader(uint vertexID [[ vertex_id ]],
         constant AAPLVertex *vertexArray [[ buffer(AAPLVertexInputIndexVertices) ]],
         constant vector_uint2 *viewportSizePointer  [[ buffer(AAPLVertexInputIndexViewportSize) ]])

{
    RasterizerData out;

    float2 pixelSpacePosition = vertexArray[vertexID].position.xy;

    float2 viewportSize = float2(*viewportSizePointer);

    out.position = vector_float4(0.0, 0.0, 0.0, 1.0);
    out.position.xy = pixelSpacePosition / (viewportSize / 2.0);

    out.textureCoordinate = vertexArray[vertexID].textureCoordinate;

    return out;
}

// Fragment function
fragment float4
samplingShader(RasterizerData in [[stage_in]],
           texture2d<half> colorTexture [[ texture(AAPLTextureIndexBaseColor) ]])
{
    constexpr sampler textureSampler (mag_filter::linear,
                                  min_filter::linear);

    // Sample the texture to obtain a color
    const half4 colorSample = colorTexture.sample(textureSampler, in.textureCoordinate);

    // return the color of the texture
    return float4(colorSample);
}

Solution

  • Per suggestion from @trojanfoe, the debugger has some useful tools for debugging GPU functions. To access this, while running your app, in XCode go to Debug > Capture GPU Frame. You can then explore the hierarchy, even see how the vertices have been loaded in etc.

    In my case, the vertices were wrong. I was providing them as pixel coordinates, rather than the -1.0, 1.0 texture coordinate system. This was evident when my verticies data in the debugger were not looking as expected. The issue for you may be something else, but it is a good place to start.

    Revised code for my issue:

    var logoQuadVertices:[AAPLVertex] = [
            //Triangle1: top left, top right, bottom left
            AAPLVertex(position: simd_float2(x: -1.0, y: -1.0), textureCoordinate: simd_float2(x: 0.0, y: 0.0)),
            AAPLVertex(position: simd_float2(x: 1.0, y: -1.0), textureCoordinate: simd_float2(x: 1.0, y: 0.0)),
            AAPLVertex(position: simd_float2(x: -1.0, y: 1.0), textureCoordinate: simd_float2(x: 0.0, y: 1.0)),
            //Triangle2: bottom left, bottom right, top right
            AAPLVertex(position: simd_float2(x: -1.0, y: 1.0), textureCoordinate: simd_float2(x: 0.0, y: 1.0)),
            AAPLVertex(position: simd_float2(x: 1.0, y: 1.0), textureCoordinate: simd_float2(x: 1.0, y: 1.0)),
            AAPLVertex(position: simd_float2(x: 1.0, y: -1.0), textureCoordinate: simd_float2(x: 1.0, y: 0.0))
    ]