MTKView displaying camera feed with lower resolution than AVCaptureVideoPreviewLayer

Question

I'm trying to stream the camera feed into a MTKView for applying some CI filter to the live stream. After initializing the capture session and having layout the MTKView, here is how I set metal (metalView is the MTKview):

func setupMetal(){
    
    metalDevice = MTLCreateSystemDefaultDevice()
    metalView.device = metalDevice
    // Write when asked
    metalView.isPaused = true
    metalView.enableSetNeedsDisplay = false
    // Command queue for the GPU
    metalCommandQueue = metalDevice.makeCommandQueue()
    // Assign the delegate
    metalView.delegate = self
    // ???
    metalView.framebufferOnly = false
}

I then grab frames from the SampleBufferDelegate and get a CIImage

extension ViewController: AVCaptureVideoDataOutputSampleBufferDelegate {

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    //try and get a CVImageBuffer out of the sample buffer
    guard let cvBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
        return
    }
    
    //get a CIImage out of the CVImageBuffer
    let ciImage = CIImage(cvImageBuffer: cvBuffer)
    
    self.currentCIImage = ciImage
    
    // We draw to the metal view everytime we receive a frame
    metalView.draw()
            
}}

I then use the currentCIImage to draw in the MTKView using its delegate methods:

extension ViewController : MTKViewDelegate {

func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) {
    //tells us the drawable's size has changed
}

func draw(in view: MTKView) {
    //create command buffer for ciContext to use to encode it's rendering instructions to the GPU
    guard let commandBuffer = metalCommandQueue.makeCommandBuffer() else {
        return
    }
    
    //make sure we actually have a ciImage to work with
    guard let ciImage = currentCIImage else {
        return
    }
    
    //make sure the current drawable object for this metal view is available (it's not in use by the previous draw cycle)
    guard let currentDrawable = view.currentDrawable else {
        return
    }
    
    //render into the metal texture
    // Check here if we find a more elegant solution for the bounds
    self.ciContext.render(ciImage,
                          to: currentDrawable.texture,
                          commandBuffer: commandBuffer,
                          bounds: CGRect(origin: .zero, size: view.drawableSize),
                          colorSpace: CGColorSpaceCreateDeviceRGB())
    
    //register where to draw the instructions in the command buffer once it executes
    commandBuffer.present(currentDrawable)
    //commit the command to the queue so it executes
    commandBuffer.commit()
}
}

It works fine and I'm able to get frames from the camera rendered in the MTKView. However, I noticed that I'm not getting the full resolution, somehow the image is zoomed in the MTKview. I know it is not an issue related to how I set the capture session because when I use the standard AVCapturePreviewLayer it is all fine. Any ideas on what I'm doing wrong?

Thanks a lot in advance!

P.S This code is mainly based on this excellent tutorial: https://betterprogramming.pub/using-cifilters-metal-to-make-a-custom-camera-in-ios-c76134993316 but somehow it doesn't seem to work for me.

Answer 1

Depending on the setup of the capture session, the camera frames will not have the same size as your MTKView. That means you need to scale and translate them before rendering to match the size of the currentDrawable. I use the following code for that (inside draw, just before the render call):

    // scale to fit into view
    let drawableSize = self.drawableSize
    let scaleX = drawableSize.width / input.extent.width
    let scaleY = drawableSize.height / input.extent.height
    let scale = min(scaleX, scaleY)
    let scaledImage = input.transformed(by: CGAffineTransform(scaleX: scale, y: scale))

    // center in the view
    let originX = max(drawableSize.width - scaledImage.extent.size.width, 0) / 2
    let originY = max(drawableSize.height - scaledImage.extent.size.height, 0) / 2
    let centeredImage = scaledImage.transformed(by: CGAffineTransform(translationX: originX, y: originY))