Transform wrong when using both AVComposition and AVVideoComposition

I am creating an AVMutableComposition. I need the asset to be flipped horizontally, so I am setting the transform of the composition track like this:

compositionTrack.preferredTransform = assetTrack.preferredTransform.scaledBy(x: -1, y: 1)

If I export this (I use AVAssetPreset960x640 as my preset), this works as expected.

However, I also need to add an AVMutableVideoComposition overlay to be rendered with this copmosition. This overlay shouldn't be flipped horizontally. I specify this like so:

// Create video composition
let videoComposition = AVMutableVideoComposition()
videoComposition.renderSize = videoSize
videoComposition.frameDuration = CMTime(value: 1, timescale: 30)
videoComposition.animationTool = AVVideoCompositionCoreAnimationTool(
    postProcessingAsVideoLayer: videoLayer,
    in: outputLayer
)

let instruction = AVMutableVideoCompositionInstruction()
instruction.timeRange = CMTimeRange(
  start: .zero,
  duration: composition.duration
)
let layerInstruction = AVMutableVideoCompositionLayerInstruction(assetTrack: compositionTrack)
layerInstruction.setTransform(assetTrack.preferredTransform, at: .zero)
    
instruction.layerInstructions = [layerInstruction]
videoComposition.instructions = [instruction]

When I export the video with this, it isn't flipped horizontally. It stops the preferredTransform applied to the composition from being performed. Presumably it's this line that is causing this:

layerInstruction.setTransform(assetTrack.preferredTransform, at: .zero)

If I set the layer transform to instead be assetTrack.preferredTransform.scaledBy(x: -1, y: 1) or compositionTrack.preferredTransform, I'm presented with a black screen on export.

Apple have these docs explaining transforms on an AVVideoComposition. If I understand correctly, they say that I should just be setting the transform of the layer instruction. If I do that – applying a transform to the layer instruction and not the composition - I'm still presented with a black screen.

Why is this and how can I fix this?

Solution

Edit: the transform that worked was

CGAffineTransform(a: -1.0, b: 0.0, c: 0.0, d: 1.0, tx: videoSize.width, ty: 0.0)

So I ran into this issue a while back, following this RayWenderlich tutorial (your code looks extremely familiar too!). The problem was that assetTrack.preferredTransform wasn't reliable, on some videos it would work, and on others it would be a black screen.

Instead of

let layerInstruction = AVMutableVideoCompositionLayerInstruction(assetTrack: compositionTrack)
layerInstruction.setTransform(assetTrack.preferredTransform, at: .zero)

Try this:

func compositionLayerInstruction(for track: AVCompositionTrack, assetTrack: AVAssetTrack, orientation: UIImage.Orientation) -> AVMutableVideoCompositionLayerInstruction {

    let instruction = AVMutableVideoCompositionLayerInstruction(assetTrack: track)
    
    var transform = CGAffineTransform.identity
    let assetSize = assetTrack.naturalSize
    
    /// you should be able to play with these values to make a horizontal flip (try changing `a` and `d`)
    switch orientation {
    case .up:
        transform = CGAffineTransform(a: 1, b: 0, c: 0, d: 1, tx: 0, ty: 0)
    case .down:
        transform = CGAffineTransform(a: -1, b: 0, c: 0, d: -1, tx: assetSize.width, ty: assetSize.height)
    case .left:
        transform = CGAffineTransform(a: 0, b: -1, c: 1, d: 0, tx: 0, ty: assetSize.width)
    case .right:
        transform = CGAffineTransform(a: 0, b: 1, c: -1, d: 0, tx: assetSize.height, ty: 0)
     default:
        print("Unsupported orientation")
    }
    
    instruction.setTransform(transform, at: .zero)
    
    return instruction
}

let layerInstruction = compositionLayerInstruction(
    for: compositionTrack, assetTrack: assetTrack, orientation: videoInfo.orientation)
/// assetTrack is the original video

I based this off another answer on SO, can't remember where though.

Here's a helper function, to get orientation

/// adjust the video orientation is the source has a different orientation
private func orientation(from transform: CGAffineTransform) -> (orientation: UIImage.Orientation, isPortrait: Bool) {
    var assetOrientation = UIImage.Orientation.up
    var isPortrait = false
    if transform.a == 0 && transform.b == 1.0 && transform.c == -1.0 && transform.d == 0 {
        assetOrientation = .right
        isPortrait = true
    } else if transform.a == 0 && transform.b == -1.0 && transform.c == 1.0 && transform.d == 0 {
        assetOrientation = .left
        isPortrait = true
    } else if transform.a == 1.0 && transform.b == 0 && transform.c == 0 && transform.d == 1.0 {
        assetOrientation = .up
    } else if transform.a == -1.0 && transform.b == 0 && transform.c == 0 && transform.d == -1.0 {
        assetOrientation = .down
    }
    return (assetOrientation, isPortrait)
}

Full usage:

/// get adjusted orientation of the video
let videoInfo = orientation(from: assetTrack.preferredTransform)
let videoSize: CGSize

if videoInfo.isPortrait {
    videoSize = CGSize(
    width: assetTrack.naturalSize.height,
    height: assetTrack.naturalSize.width)
} else {
    videoSize = assetTrack.naturalSize
}


 /// the video
let videoLayer = CALayer()
videoLayer.frame = CGRect(origin: .zero, size: videoSize)
        
/// overlay where you add graphics and other stuff
let overlayLayer = CALayer()
overlayLayer.frame = CGRect(origin: .zero, size: videoSize)

let outputLayer = CALayer()
outputLayer.frame = CGRect(origin: .zero, size: videoSize)
outputLayer.addSublayer(videoLayer)
outputLayer.addSublayer(overlayLayer)

let videoComposition = AVMutableVideoComposition()
videoComposition.renderSize = videoSize
videoComposition.frameDuration = CMTime(value: 1, timescale: 30)
videoComposition.animationTool = AVVideoCompositionCoreAnimationTool(
postProcessingAsVideoLayer: videoLayer,
in: outputLayer)

// MARK: - previous `let layerInstruction` code goes here...

instruction.layerInstructions = [layerInstruction]

/// export session code here...