Search code examples
hololensmrtkmr-spectatorview

MRTK new spectator view: How to instantiate a prefab at runtime on both server and client?

The new spectator view seems to have changed the way it syncs states. It doesn't use UNet anymore but each component has a state synchronization service.

I can see how this synchronizes changes in objects that were in the scene from the start, but how does it work if I want to instantiate a prefab during runtime?

In the old UNet implementation I would use the NetworkManager to instantiate it on all connected devices, and the prefab would be created everywhere with all its referenes to child components intact. What is the new way of doing this?

Solution

  • State synchronization logic for Spectator View should automatically assess game object creation on the HoloLens/Broadcaster device and relay creation events to spectator/Observer devices. Spectator View logic does not support an object created on the spectator device being recreated on the HoloLens device. If this aligns with your needs, the flow of logic is the following:

    1. When a game object (flagged for synchronizing) is declared in the HoloLens's scene, it gets a TransformBroadcaster component added. This TransformBroadcaster monitors child hierarchy changes, so when a child game object of this TransformBroadcaster's game object is created, the parent TransformBroadcaster adds a TransformBroadcaster component to the child game object: https://github.com/microsoft/MixedReality-SpectatorView/blob/73414f4429048c6f0eed154fb8bbdea93b4caa81/src/SpectatorView.Unity/Assets/SpectatorView/Scripts/StateSynchronization/TransformBroadcaster.cs#L576

    2. Each frame update, the TransformBroadcaster also observes changes to its game object. If new components have been added to the game object that have associated ComponentBroadcasterServices, it adds ComponentBroadcasters to the game object: https://github.com/microsoft/MixedReality-SpectatorView/blob/73414f4429048c6f0eed154fb8bbdea93b4caa81/src/SpectatorView.Unity/Assets/SpectatorView/Scripts/StateSynchronization/TransformBroadcaster.cs#L502

    3. Once a ComponentBroadcaster (including the TransformBroadcaster) is added to a game object, it will assess all spectators on frame updates. If the ComponentBroadcaster detects that there is a spectator that does not know of its existence, it will send said spectator a broadcaster creation event: https://github.com/microsoft/MixedReality-SpectatorView/blob/73414f4429048c6f0eed154fb8bbdea93b4caa81/src/SpectatorView.Unity/Assets/SpectatorView/Scripts/StateSynchronization/ComponentBroadcaster.cs#L177

    4. The creation event will be received by the spectator device in the StateSynchronizationSceneManager. When the StateSynchronizationSceneManager sees this creation event, it obtains or creates a mirror game object (essentially a copy of the game object running on the HoloLens device). It then adds the corresponding ComponentObserver to the mirror game object using the associated IComponentBroadcasterService: https://github.com/microsoft/MixedReality-SpectatorView/blob/73414f4429048c6f0eed154fb8bbdea93b4caa81/src/SpectatorView.Unity/Assets/SpectatorView/Scripts/StateSynchronization/StateSynchronizationSceneManager.cs#L241

    5. When a ComponentObserver is added to a game object, it creates the component it is observing: https://github.com/microsoft/MixedReality-SpectatorView/blob/73414f4429048c6f0eed154fb8bbdea93b4caa81/src/SpectatorView.Unity/Assets/SpectatorView/Scripts/StateSynchronization/ComponentObserver.cs#L27 Then with each payload update, the ComponentObserver updates this attached component to mimic what is being displayed on the HoloLens device.

    Once all ComponentBroadcasters register as created with the spectator device, you should have your newly created game object in the HoloLens scene recreated in the spectator scene; however, there a few other things to call out:

    1. Game objects need to be flagged in some manner as candidates for synchronization. This can be achieved through two manners. You can declare that all objects should be synchronized in your scene through the Spectator View settings: https://github.com/microsoft/MixedReality-SpectatorView/blob/master/doc/SpectatorView.Setup.md#hololens-2--hololens Or you can add a GameObjectHierarchyBroadcaster to your HoloLens scene (All child game objects under a GameObjectHierarchyBroadcaster component will be synchronized).

    2. If you have custom Unity types, you may need to create your own IComponentBroadcasterService, IComponentBroadcaster and IComponentObserver. Most of the default Unity components (transforms, meshes, etc) already have component broadcaster services, but your own custom types may require additional work.

    3. You need to have the same Asset Caches compiled into the HoloLens and spectator applications. When ComponentBroadcasters are created on spectator devices, they are often handed assetIds that are used to instantiate mirror game object components. For example, when a new MeshFilter is created it uses an assetId to look up the MeshFilter through the AssetService to know what mesh to add to the mirror game object: https://github.com/microsoft/MixedReality-SpectatorView/blob/73414f4429048c6f0eed154fb8bbdea93b4caa81/src/SpectatorView.Unity/Assets/SpectatorView/Scripts/StateSynchronization/NetworkedComponents/MeshFilter/MeshFilterObserver.cs#L15 This means that dynamically creating a mesh compared to relying on a mesh defined in one of your Assets will not work correctly.