Creating AVComposition with a Blurry Background in iOS

I'm not really sure how to go about asking this question, I would appreciate any helpful feedback on improving this question. I am trying to make a function that accepts a video URL as input (local video), in turn this function tries to create a video with a blurry background, with the original video at its center and scaled down. My issue is that my code is working fine, aside from when I use videos that are directly recorded from the iPhone camera.

An example of what I am trying to achieve is the following (Taken from my code):

The input video here is an mp4. I have been able to make the code work as well with mov files that I've downloaded online. But when I use mov files recorded from the iOS camera, I end up with the following:

(How can i post pictures that take less space in the question?)

Now, the reason I am not sure how to ask this question is because there is a fair amount of code in the process and I haven't been able to fully narrow down the question but I believe it is in the function that I will paste below. I will also post a link to a github repository, where a barebones version of my project has been posted for anyone curious or willing to help. I must confess that the code I am using was originally written by a StackOverflow user named TheTiger on the following question: AVFoundation - Adding blur background to video . I've refactored segments of this, and with their permission, was allowed to post the question here.

My github repo is linked here: GITHUB REPO My demo is set up with 3 different videos, an mp4 downloaded from the web (working), an mov downloaded from the web (Working) and an mov I've recorded on my phone (not working)

The code I imagine is causing the issue is here:

fileprivate func addAllVideosAtCenterOfBlur(asset: AVURLAsset, blurVideo: AVURLAsset, scale: CGFloat, completion: @escaping BlurredBackgroundManagerCompletion) {

    let mixComposition = AVMutableComposition()

    var instructionLayers : Array<AVMutableVideoCompositionLayerInstruction> = []

    let blurVideoTrack = mixComposition.addMutableTrack(withMediaType: AVMediaType.video, preferredTrackID: kCMPersistentTrackID_Invalid)

    if let videoTrack = blurVideo.tracks(withMediaType: AVMediaType.video).first {
        let timeRange = CMTimeRange(start: .zero, duration: blurVideo.duration)
        try? blurVideoTrack?.insertTimeRange(timeRange, of: videoTrack, at: .zero)
    }

    let timeRange = CMTimeRange(start: .zero, duration: asset.duration)

    let track = mixComposition.addMutableTrack(withMediaType: AVMediaType.video, preferredTrackID: kCMPersistentTrackID_Invalid)

    if let videoTrack = asset.tracks(withMediaType: AVMediaType.video).first {

        try? track?.insertTimeRange(timeRange, of: videoTrack, at: .zero)
        let layerInstruction = AVMutableVideoCompositionLayerInstruction(assetTrack: track!)

        let properties = scaleAndPositionInAspectFitMode(forTrack: videoTrack, inArea: size, scale: scale)

        let videoOrientation = videoTrack.getVideoOrientation()
        let assetSize = videoTrack.assetSize()

        let preferredTransform = getPreferredTransform(videoOrientation: videoOrientation, assetSize: assetSize, defaultTransform: asset.preferredTransform, properties: properties)

        layerInstruction.setTransform(preferredTransform, at: .zero)

        instructionLayers.append(layerInstruction)
    }

    /// Adding audio
    if let audioTrack = asset.tracks(withMediaType: AVMediaType.audio).first {
        let aTrack = mixComposition.addMutableTrack(withMediaType: AVMediaType.audio, preferredTrackID: kCMPersistentTrackID_Invalid)
        try? aTrack?.insertTimeRange(timeRange, of: audioTrack, at: .zero)
    }


    /// Blur layer instruction
    let layerInstruction = AVMutableVideoCompositionLayerInstruction(assetTrack: blurVideoTrack!)
    instructionLayers.append(layerInstruction)

    let mainInstruction = AVMutableVideoCompositionInstruction()
    mainInstruction.timeRange = timeRange
    mainInstruction.layerInstructions = instructionLayers

    let mainCompositionInst = AVMutableVideoComposition()
    mainCompositionInst.instructions = [mainInstruction]
    mainCompositionInst.frameDuration = CMTimeMake(value: 1, timescale: 30)
    mainCompositionInst.renderSize = size

    //let url = URL(fileURLWithPath: "/Users/enacteservices/Desktop/final_video.mov")
    let url = self.videoOutputUrl(filename: "finalBlurred")
    try? FileManager.default.removeItem(at: url)

    performExport(composition: mixComposition, instructions: mainCompositionInst, stage: 2, outputUrl: url) { (error) in
        if let error = error {
            completion(nil, error)
        } else {
            completion(url, nil)
        }
    }
}

The getPreferredTransform() function is also quite relevant:

fileprivate func getPreferredTransform(videoOrientation: UIImage.Orientation, assetSize: CGSize, defaultTransform: CGAffineTransform, properties: Properties) -> CGAffineTransform {
    switch videoOrientation {
    case .down:
        return handleDownOrientation(assetSize: assetSize, defaultTransform: defaultTransform, properties: properties)
    case .left:
        return handleLeftOrientation(assetSize: assetSize, defaultTransform: defaultTransform, properties: properties)
    case .right:
        return handleRightOrientation(properties: properties)
    case .up:
        return handleUpOrientation(assetSize: assetSize, defaultTransform: defaultTransform, properties: properties)
    default:
        return handleOtherCases(assetSize: assetSize, defaultTransform: defaultTransform, properties: properties)
    }
}

fileprivate func handleDownOrientation(assetSize: CGSize, defaultTransform: CGAffineTransform, properties: Properties) -> CGAffineTransform {
    let rotateTransform = CGAffineTransform(rotationAngle: -CGFloat(Double.pi/2.0))

    // Scale
    let scaleTransform = CGAffineTransform(scaleX: properties.scale.width, y: properties.scale.height)

    // Translate
    var ytranslation: CGFloat = assetSize.height
    var xtranslation: CGFloat = 0
    if properties.position.y == 0 {
        xtranslation = -(assetSize.width - ((size.width/size.height) * assetSize.height))/2.0
    }
    else {
        ytranslation = assetSize.height - (assetSize.height - ((size.height/size.width) * assetSize.width))/2.0
    }
    let translationTransform = CGAffineTransform(translationX: xtranslation, y: ytranslation)

    // Final transformation - Concatination
    let finalTransform = defaultTransform.concatenating(rotateTransform).concatenating(translationTransform).concatenating(scaleTransform)
    return finalTransform
}

fileprivate func handleLeftOrientation(assetSize: CGSize, defaultTransform: CGAffineTransform, properties: Properties) -> CGAffineTransform {

    let rotateTransform = CGAffineTransform(rotationAngle: -CGFloat(Double.pi))

    // Scale
    let scaleTransform = CGAffineTransform(scaleX: properties.scale.width, y: properties.scale.height)

    // Translate
    var ytranslation: CGFloat = assetSize.height
    var xtranslation: CGFloat = assetSize.width
    if properties.position.y == 0 {
        xtranslation = assetSize.width - (assetSize.width - ((size.width/size.height) * assetSize.height))/2.0
    } else {
        ytranslation = assetSize.height - (assetSize.height - ((size.height/size.width) * assetSize.width))/2.0
    }
    let translationTransform = CGAffineTransform(translationX: xtranslation, y: ytranslation)

    // Final transformation - Concatination
    let finalTransform = defaultTransform.concatenating(rotateTransform).concatenating(translationTransform).concatenating(scaleTransform)

    return finalTransform
}

fileprivate func handleRightOrientation(properties: Properties) -> CGAffineTransform  {
    let scaleTransform = CGAffineTransform(scaleX: properties.scale.width, y: properties.scale.height)

    // Translate
    let translationTransform = CGAffineTransform(translationX: properties.position.x, y: properties.position.y)

    let finalTransform  = scaleTransform.concatenating(translationTransform)
    return finalTransform
}

fileprivate func handleUpOrientation(assetSize: CGSize, defaultTransform: CGAffineTransform, properties: Properties) -> CGAffineTransform {

    return handleOtherCases(assetSize: assetSize, defaultTransform: defaultTransform, properties: properties)
}

fileprivate func handleOtherCases(assetSize: CGSize, defaultTransform: CGAffineTransform, properties: Properties) -> CGAffineTransform {
    let rotateTransform = CGAffineTransform(rotationAngle: CGFloat(Double.pi/2.0))

    let scaleTransform = CGAffineTransform(scaleX: properties.scale.width, y: properties.scale.height)

    var ytranslation: CGFloat = 0
    var xtranslation: CGFloat = assetSize.width
    if properties.position.y == 0 {
        xtranslation = assetSize.width - (assetSize.width - ((size.width/size.height) * assetSize.height))/2.0
    }
    else {
        ytranslation = -(assetSize.height - ((size.height/size.width) * assetSize.width))/2.0
    }
    let translationTransform = CGAffineTransform(translationX: xtranslation, y: ytranslation)

    let finalTransform = defaultTransform.concatenating(rotateTransform).concatenating(translationTransform).concatenating(scaleTransform)
    return finalTransform
}

Solution

The problem is with your handleOtherCases function in which you create and return CGAffineTransform to be applied to the frame. The scale and rotation transform applied are fine. But the translation transform you have calculated is not correct. Do try the code snippet below and it would produce the desired result as in the image attached.

    fileprivate func handleOtherCases(assetSize: CGSize, defaultTransform: CGAffineTransform, properties: Properties) -> CGAffineTransform {
        let rotateTransform = CGAffineTransform(rotationAngle: CGFloat(Double.pi/2.0))

        let scaleTransform = CGAffineTransform(scaleX: properties.scale.width, y: properties.scale.height)

        let ytranslation: CGFloat = ( self.size.height - ( assetSize.height * properties.scale.height ) ) / 2
        let xtranslation: CGFloat = ( assetSize.width * properties.scale.width ) + ( self.size.width - ( assetSize.width * properties.scale.width ) ) / 2
        let translationTransform = CGAffineTransform(translationX: xtranslation, y: ytranslation)

        let finalTransform = defaultTransform.concatenating(scaleTransform).concatenating(rotateTransform).concatenating(translationTransform)
        return finalTransform
    }