No rendering with AVMutableVideoComposition

Question

I have 3 videos that I am sequencing using an AVMutableComposition and then playing the video using an AVPlayer and grabbing the frames using an AVPlayerItemVdeoOutput. The video sequence is as follows:

[Logo Video - n seconds][Main video - m seconds][Logo Video - l seconds]

The code looks like this:

    // Build the composition.
    pComposition            = [AVMutableComposition composition];

    // Fill in the assets that make up the composition
    AVMutableCompositionTrack* pCompositionVideoTrack   = [pComposition addMutableTrackWithMediaType: AVMediaTypeVideo preferredTrackID: 1];
    AVMutableCompositionTrack* pCompositionAudioTrack   = [pComposition addMutableTrackWithMediaType: AVMediaTypeAudio preferredTrackID: 2];

    CMTime          time            = kCMTimeZero;
    CMTimeRange     keyTimeRange    = kCMTimeRangeZero;
    for( AVAsset* pAssetsAsset in pAssets )
    {
        AVAssetTrack*   pAssetsAssetVideoTrack  = [pAssetsAsset tracksWithMediaType: AVMediaTypeVideo].firstObject;
        AVAssetTrack*   pAssetsAssetAudioTrack  = [pAssetsAsset tracksWithMediaType: AVMediaTypeAudio].firstObject;

        CMTimeRange     timeRange               = CMTimeRangeMake( kCMTimeZero, pAssetsAsset.duration );

        NSError*        pVideoError             = nil;
        NSError*        pAudioError             = nil;

        if ( pAssetsAssetVideoTrack )
        {
            [pCompositionVideoTrack insertTimeRange: timeRange ofTrack: pAssetsAssetVideoTrack atTime: time error: &pVideoError];
        }
        if ( pAssetsAssetAudioTrack )
        {
            [pCompositionAudioTrack insertTimeRange: timeRange ofTrack: pAssetsAssetAudioTrack atTime: time error: &pAudioError];
        }

        if ( pAssetsAsset == pKeyAsset )
        {
            keyTimeRange    = CMTimeRangeMake( time, timeRange.duration );
        }

        NSLog( @"%@", [pVideoError description] );
        NSLog( @"%@", [pAudioError description] );

        time        = CMTimeAdd( time, pAssetsAsset.duration );
    }

The logo videos are silent and merely display my logo. I manually create these videos so everything is perfect here. The "Main Video" however can end up with the wrong orientation. To combat this an AVMutableVideoComposition looks like the perfect way forward. So I setup a simple video composition that does a simple setTransform as follows:

pAsset = pComposition; pPlayerItem = [AVPlayerItem playerItemWithAsset: pAsset]; pPlayer = [AVPlayer playerWithPlayerItem: pPlayerItem];

    NSArray*        pPlayerTracks   = [pAsset tracksWithMediaType: AVMediaTypeVideo];
    AVAssetTrack*   pPlayerTrack    = pPlayerTracks[0];

    pVideoCompositionLayerInstruction               = [AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstruction];
    [pVideoCompositionLayerInstruction setTransform: [[pKeyAsset tracksWithMediaType: AVMediaTypeVideo].firstObject preferredTransform] atTime: kCMTimeZero];

    pVideoCompositionInstruction                    = [AVMutableVideoCompositionInstruction videoCompositionInstruction];
    pVideoCompositionInstruction.backgroundColor    = [[UIColor blackColor] CGColor];
    pVideoCompositionInstruction.timeRange          = keyTimeRange;
    pVideoCompositionInstruction.layerInstructions  = @[ pVideoCompositionLayerInstruction ];

    pVideoComposition                               = [AVMutableVideoComposition videoComposition];
    pVideoComposition.renderSize                    = [[pKeyAsset tracksWithMediaType: AVMediaTypeVideo].firstObject naturalSize];
    pVideoComposition.frameDuration                 = [[pKeyAsset tracksWithMediaType: AVMediaTypeVideo].firstObject minFrameDuration];
    pVideoComposition.instructions                  = @[ pVideoCompositionInstruction ];

    pPlayerItem.videoComposition    = pVideoComposition;

However when I come to play the video sequence I get no output returned. AVPlayerItemVideoOutput hasNewPixelBufferForItemTime always returns NO. If I comment out the last line in the code above (ie the setting the videoComposition) then everything works as before (with videos with the wrong orientation). Does anybody know what I'm doing wrong? Any thoughts much appreciated!

Answer 1

The issue here is that keyTimeRange may not start at time zero if your Logo video has nonzero duration. pVideoCompositionInstruction will start at keyTimeRange.start, rather than kCMTimeZero (when the AVMutableComposition will start), which violates the rules for AVVideoCompositionInstructions

"For the first instruction in the array, timeRange.start must be less than or equal to the earliest time for which playback or other processing will be attempted (typically kCMTimeZero)", according to the docs

To solve this, set pVideoComposition.instructions to an array containing three AVMutableVideoCompositionInstruction objects, each with their own AVMutableVideoCompositionLayerInstruction according to each AVAsset's transform. The time range for each of the three instructions should be the times at which these assets appear in the composition track. Make sure they line up exactly.