Swift/SwiftUI/AVFoundation: Is there a way to take the "output" of two AVPlayers and combine/composite that in a new AVPlayer?

Question

Say you have two (already configured) AVPlayer instances (i.e., their playerItems are loaded with some AVAsset, and presented so that you can play/pause, etc.)

I was wondering if there is any way to take whatever each of those players is showing (that is, their "output" if you will) and compose those as two "layers" in another, separate AVPlayer in a way that if, for example, you pause one of the "source" players that layer also pauses in the composited player, or if you apply an effect to one of the sources, that effect will also reflect in the composited AVPlayer?

I have been looking into CADisplayLink and am wondering if that is maybe an option - I should say that my experience with AVFoundation and its related APIs is very limited, hence my question.

Thanks in advance.

Answer 1

Just in case anyone finds this, here's how I ended up solving this need:

1. CADisplayLink has, of course, nothing to do with it - I thought the name suggested something that it isn't; CADisplayLink is, for all intents and purposes, a timer that is "linked" to the display refresh rate - terrifically useful, just not for what I needed;
2. Since what I needed was to display exactly what both players were outputting, including any processing and filters applied to the media, I also looked into AVVideoComposition, and although that could be used to achieve something like this for a finite media (i.e., for a specific length of time), but it did not work for my application, which requires continuous display - think of it as an infinite-length media, like live-streaming, but it is not live streaming;
3. After some fighting with concepts and ideas, I realised that I was going at this the wrong way, and that the solution is actually quite simple, come to think of it.

Here's my solution (keep in mind that I am working on a SwiftUI lifetime application, hence some of the choices):

• I ended up creating a UIViewRepresentable that creates an instance of a custom UIKit view - let's call it CompositePlayer;
• This is a custom UIView (a subclass, actually) that receives two AVQueuePlayer objects as parameters (these are the players where I load the media, apply filters, etc.) - let's call them player_1 and player_2 (I also declared these as var although I am sure you don't need to if you're not planning on changing the object later - I did);
• The class instantiates two "fresh" AVPlayerLayer objects (playerLayer_1 and playerLayer_2 respectively)
• Those two received players are then assigned as the respective AVPlayerLayer.player property;
• And then are added as subLayers to the view layer in the desired order (i.e., player_1 first, then player_2 which makes it render "on top" of player_1)

Pseudo-code(ish) would be something like:

class CompositePlayer: UIView {

    var player_1: AVQueuePlayer
    var player_2: AVQueuePlayer

    private let playerLayer_1: AVPlayerLayer()
    private let playerLayer_2: AVPlayerLayer()

    // You need to provide any initialiser - it is not mandatory
    // to override the default init(frame: CGRect); I'm sure you
    // knew that, but n00bs like me don't, really...

    init(frame: CGRect, player_1: AVQueuePlayer, player_2: AVQueuePlayer) {
        self.player_1 = player_1
        self.player_2 = player_2
        super.init(frame: frame)

        // Now that we are initialised, assign the players to the
        // respective player layers...
        self.playerLayer_1.player = player_1
        self.playerLayer_2.player = player_2

        // ... and add these to the view's layer as sublayers in
        // the desired order - here, player_2 will render "on top"
        // of player_1:
        layer.addSublayer(playerLayer_1)
        layer.addSubLayer(playerLayer_2)
    }

    // This is required (as noted by the keyword...) when subclassing
    // UIView
    required init?(coder: NSCoder) {
        fatalError("init(coder:) has not been implemented")
    }

    // Finally, you need to set the frame size for each layer, which
    // is easily achievable by overriding the layoutSubviews() method:
    override func layoutSubviews() {
        super.layoutSubviews()
        playerLayer_1.frame = bounds
        playerLayer_2.frame = bounds
    }
}

What do you get with this?

You get a view that you then instantiate, as previously said, by ways of an UIViewRepresentable that is able to render exactly the same thing as you regular player is rendering in its own "original" instance (warts and all), since AVQueuePlayer objects can effectively output to more than one view; and if you properly assign the right objects to members of the UIViewRepresentable struct (by using, say, an EnvironmentObject), you get the additional bonus of being able to access both playerLayer objects elsewhere and of being able to set, for example, that layer's opacity ,transform it, etc.

And as if by magic, whatever operation you make on your "original" players will reflect 100% on this view - with some notable exceptions inherent to the view and not the player: for example, setting the videoGravity of a player will not affect this view's display, since you won't be "operating" on its geometry.

Phew. That was long (and I really shortened it!). I hope it helps somebody!