Tracking multiple images and play their videos in ARkit2

Question

I've got below sample code, tracking a single image then plays a video using Apple's ARkit. Currently videos stored locally on the device. The code below just tracks a single image and plays corresponding video. How can I modify it and let it track multiple images and play their videos?

 @IBOutlet var sceneView: ARSCNView!

// Create video player
let videoPlayer: AVPlayer = {
    // Load cat video from bundle
    guard let url = Bundle.main.url(forResource: "nyan-cat", withExtension: "mp4", subdirectory: "art.scnassets") else {
        print("Could not find video file")
        return AVPlayer()
    }

    let player = AVPlayer(url: url)

    NotificationCenter.default.addObserver(forName: NSNotification.Name.AVPlayerItemDidPlayToEndTime, object: nil, queue: nil) { notification in
        player.seek(to: CMTime.zero)
        player.play()
    }
    return player
}()

func renderer(_ renderer: SCNSceneRenderer, nodeFor anchor: ARAnchor) -> SCNNode? {
    let node = SCNNode()

    // Show video overlayed to image
    if let imageAnchor = anchor as? ARImageAnchor {

        // Create a plane
        let plane = SCNPlane(width: imageAnchor.referenceImage.physicalSize.width,
                             height: imageAnchor.referenceImage.physicalSize.height)

        // Set AVPlayer as the plane's texture and play
        plane.firstMaterial?.diffuse.contents = self.videoPlayer
        self.videoPlayer.play()

        let playNode = SCNNode(geometry: plane)

        // Rotate the plane to match the anchor
        playNode.eulerAngles.x = -.pi / 2

        // Add plane node to parent
        node.addChildNode(playNode)
    }

    return node
}

Answer 1

As @SilentK has stated, in order to track multiple ARImageAnchors, you need to set the maximum number of images to track e.g:

configuration.maximumNumberOfTrackedImages = 10

Which as of IOS12 is available for both ARWorldTrackingConfiguration and ARImageTrackingConfiguration.

As you will also know that when you add an ARReferenceImage to your Assets Bundle, you have to assign it a name e.g:

Now since each target contains a unique name, you can easily use this to load your different videos. So it makes most sense to name each video the same as your imageTarget.

An example of what you are attempting to do might look like so:

//-------------------------
//MARK: - ARSCNViewDelegate
//-------------------------

extension ViewController: ARSCNViewDelegate{

    func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {

        //1. Check We Have Detected An ARImageAnchor
        guard let validAnchor = anchor as? ARImageAnchor else { return }

        //2. Create A Video Player Node For Each Detected Target
        node.addChildNode(createdVideoPlayerNodeFor(validAnchor.referenceImage))

    }


    /// Creates An SCNNode With An AVPlayer Rendered Onto An SCNPlane
    ///
    /// - Parameter target: ARReferenceImage
    /// - Returns: SCNNode
    func createdVideoPlayerNodeFor(_ target: ARReferenceImage) -> SCNNode{

        //1. Create An SCNNode To Hold Our VideoPlayer
        let videoPlayerNode = SCNNode()

        //2. Create An SCNPlane & An AVPlayer
        let videoPlayerGeometry = SCNPlane(width: target.physicalSize.width, height: target.physicalSize.height)
        var videoPlayer = AVPlayer()

        //3. If We Have A Valid Name & A Valid Video URL The Instanciate The AVPlayer
        if let targetName = target.name,
            let validURL = Bundle.main.url(forResource: targetName, withExtension: "mp4", subdirectory: "/art.scnassets") {
            videoPlayer = AVPlayer(url: validURL)
            videoPlayer.play()
        }

        //4. Assign The AVPlayer & The Geometry To The Video Player
        videoPlayerGeometry.firstMaterial?.diffuse.contents = videoPlayer
        videoPlayerNode.geometry = videoPlayerGeometry

        //5. Rotate It
        videoPlayerNode.eulerAngles.x = -.pi / 2

        return videoPlayerNode

    }

}

Infinite looping playback using this method is quite CPU intensive if you have several instances rendered:

 NotificationCenter.default.addObserver(forName: NSNotification.Name.AVPlayerItemDidPlayToEndTime, object: nil, queue: nil) { notification in
        player.seek(to: CMTime.zero)
        player.play()
    }

As such you will probably want to add logic to only play one video at a time, e.g. performing an SCNHitTest on the VideoNode, which will trigger it playing etc.

Hope it helps...