Implementing AVVideoCompositing causes video rotation problems

Question

I using Apple's example https://developer.apple.com/library/ios/samplecode/AVCustomEdit/Introduction/Intro.html and have some issues with video transformation. If source assets have preferredTransform other than identity, output video will have incorrectly rotated frames. This problem can be fixed if AVMutableVideoComposition doesn't have value in property customVideoCompositorClass and when AVMutableVideoCompositionLayerInstruction's transform is setted up with asset.preferredTransform. But in reason of using custom video compositor, which adopting an AVVideoCompositing protocol I can't use standard video compositing instructions. How can I pre-transform input asset tracks before it's CVPixelBuffer's putted into Metal shaders? Or there are any other way to fix it?

Fragment of original code:

func buildCompositionObjectsForPlayback(_ forPlayback: Bool, overwriteExistingObjects: Bool) {

    // Proceed only if the composition objects have not already been created.
    if self.composition != nil && !overwriteExistingObjects { return }
    if self.videoComposition != nil && !overwriteExistingObjects { return }

    guard !clips.isEmpty else { return }

    // Use the naturalSize of the first video track.
    let videoTracks = clips[0].tracks(withMediaType: AVMediaType.video)
    let videoSize = videoTracks[0].naturalSize

    let composition = AVMutableComposition()

    composition.naturalSize = videoSize

    /*
     With transitions:
     Place clips into alternating video & audio tracks in composition, overlapped by transitionDuration.
     Set up the video composition to cycle between "pass through A", "transition from A to B", "pass through B".
    */
    let videoComposition = AVMutableVideoComposition()

    if self.transitionType == TransitionType.diagonalWipe.rawValue {
        videoComposition.customVideoCompositorClass = APLDiagonalWipeCompositor.self
    } else {
        videoComposition.customVideoCompositorClass = APLCrossDissolveCompositor.self
    }

    // Every videoComposition needs these properties to be set:
    videoComposition.frameDuration = CMTimeMake(1, 30) // 30 fps.
    videoComposition.renderSize = videoSize

    buildTransitionComposition(composition, andVideoComposition: videoComposition)

    self.composition = composition
    self.videoComposition = videoComposition
}

UPDATE: I did workaround for transforming like this:

private func makeTransformedPixelBuffer(fromBuffer buffer: CVPixelBuffer, withTransform transform: CGAffineTransform) -> CVPixelBuffer? {
    guard let newBuffer = renderContext?.newPixelBuffer() else {
        return nil
    }

    // Correct transformation example I took from https://stackoverflow.com/questions/29967700/coreimage-coordinate-system
    var preferredTransform = transform
    preferredTransform.b *= -1
    preferredTransform.c *= -1

    var transformedImage = CIImage(cvPixelBuffer: buffer).transformed(by: preferredTransform)
    preferredTransform = CGAffineTransform(translationX: -transformedImage.extent.origin.x, y: -transformedImage.extent.origin.y)
    transformedImage = transformedImage.transformed(by: preferredTransform)

    let filterContext = CIContext(mtlDevice: MTLCreateSystemDefaultDevice()!)
    filterContext.render(transformedImage, to: newBuffer)

    return newBuffer
}

But wondering if there are more memory-effective way without creation of new pixel buffers

Answer 1

How can I pre-transform input asset tracks before it's CVPixelBuffer's putted into Metal shaders?

The best way to achieve maximum performance is to transform your video frame directly in shader. You just need to add rotation matrix in your Vertex shader.