Switching AVCaptureSessionPreset before capturing photo

Question

I'm making live camera filter app.

I use AVCaptureVideoDataOutput to pass samplebuffer to MTKView for rendering preview and AVCapturePhotoOutput for capturing photos.

And my app has some aspect ratio options for capturing photos. (including 16:9 and 4:3)

What I want to do is to make the preview in full screen size (which is 16:9 but anything closer to this would also be fine) although users select 4:3 option.

And I'm planning to show border lines inside the preview so that users can figure out the photo ouput size.

I need 16:9 preset option like 1280*720 for the preview and photo preset option for capturing photo.

I came up with few ideas.

1. Having two AVCaptureSessions with different preset --> not good for performance

2. Use 1280*720 preset for capturing and crop the photo output to 4:3 aspect ratio --> low-resolution photos

3. Switch preset just before call photoOutput.capturePhoto method. --> preview freezes in a moment because AVCaptureSession has to be updated

I decided to go with 3, but it gives me an error.

(if there is a better way, please let me know)

this is my code.

@IBAction func takePhoto(_ sender: UIButton) {
    captureSessionQueue.async {
        var photoSettings = AVCapturePhotoSettings()
        photoSettings.isHighResolutionPhotoEnabled = true

        // switch preset from .hd1280*720 to .photo
        self.session.beginConfiguration()
        if self.session.canSetSessionPreset(.photo) {
            self.session.sessionPreset = .photo
        }
        self.session.commitConfiguration()

        self.photoOutput.capturePhoto(with: photoSettings, delegate: self)

        self.session.beginConfiguration()
        self.session.sessionPreset = .hd1280*720
        self.session.commitConfiguration()
    }
}

the error is,

Error Domain=AVFoundationErrorDomain Code=-11800 "The operation could not be completed" UserInfo={NSLocalizedFailureReason=An unknown error occurred (-16800), NSLocalizedDescription=The operation could not be completed, NSUnderlyingError=0x280013720 {Error Domain=NSOSStatusErrorDomain Code=-16800 "(null)"}}

I think this is because I call capturePhoto method before the session completes the update to new preset.

when I call self.photoOutput.capturePhoto method 1 or 2 seconds after the commitConfiguration(), it works.

So, is there any way that I can know the completion of AVCaptureSession updates or is there a better solution dealing with different aspect ratio between the video data output and photo output?

Answer 1

The best way to output a different aspect ratio for video and photo is to process your photo image data in the AVCapturePhotoCaptureDelegate delegate. You could do something like the following. In the didFinishProcessingPhoto function you crop your image and create jpeg data from it. You will also need to copy the photo metadata into this jpeg data. Then in the didFinishCaptureFor function you can save this data to the Photo Library.

class MyPhotoCaptureDelegate: NSObject, AVCapturePhotoCaptureDelegate {
    func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) {
        // crop image
        if let cgImage = photo.cgImageRepresentation()?.takeUnretainedValue() {
            let cropRect : CGRect = calculateAspectRatioCrop(cgImage:cgImage, aspectRatio:16.0/9.0)
            if let cgCroppedImage = cgImage.cropping(to:cropRect) {
                let image = UIImage(cgImage:cgCroppedImage)
                if let photoData = image.jpegData(compressionQuality: 0.9) {
                    // add meta data from original image to cropped image data
                    self.photoData = addImageProperties(imageData: photoData, properties: photo.metadata as NSDictionary)
                }
            }
        }
    }

    func photoOutput(_ captureOutput: AVCapturePhotoOutput, didFinishCaptureFor resolvedSettings: AVCaptureResolvedPhotoSettings, error: Error?) {
        if let error = error {
            print("Error capturing photo: \(error)")
            return
        }

        guard let photoData = photoData else {
            print("No photo data resource")
            return
        }
        savePhotoData()
    }

    func calculateAspectRatioCrop(cgImage : CGImage, aspectRatio: CGFloat) -> CGRect {
        var width = CGFloat(cgImage.width)
        var height = CGFloat(cgImage.height)
        // should be cropped vertically or horizontally?
        if aspectRatio > width/height {
            height = width / aspectRatio
        } else {
            width = height * aspectRatio
        }
        return CGRect(x: (CGFloat(cgImage.width) - width)/2, y: (CGFloat(cgImage.height) - height)/2, width: width, height: height)
    }

    func addImageProperties(imageData: Data, properties: NSDictionary?) -> Data? {
        // create an imagesourceref
        if let source = CGImageSourceCreateWithData(imageData as CFData, nil) {
            // this is of type image
            if let uti = CGImageSourceGetType(source) {
                // create a new data object and write the new image into it
                let destinationData = NSMutableData()
                if let destination = CGImageDestinationCreateWithData(destinationData, uti, 1, nil) {
                    // add the image contained in the image source to the destination, overidding the old metadata with our modified metadata
                    CGImageDestinationAddImageFromSource(destination, source, 0, properties)
                    if CGImageDestinationFinalize(destination) == false {
                        return nil
                    }
                    return destinationData as Data
                }
            }
        }
        return nil
    }

    private var photoData : Data? = nil
}

I have left you to fill in the savePhotoData() function.