Search code examples
swiftaugmented-realityarkitrealitykit

Apply a Shader on RealityKit Scene

Is there a way to apply a full screen shader to a RealityKit scene (Swift/iOS)?

I know I can do this with SCNTechnique in SceneKit/ARSCNView, but I'm dealing with an ARView.

  • Is there a way in RealityKit one can apply a vertex/fragment shader?
  • or a way to get the rendered scene as a texture?

With an ARSCNView I can apply a shader to run over the output of a scene, just like: https://medium.com/ar-tips-and-tricks/arkit-theory-an-introduction-to-scntechniques-710e024bc91e

But with RealityKit, I'm dealing with an ARView which doesn't have a SCNTechnique property

Thanks :-)

Solution

  • iOS RealityKit

    In RealityKit for iOS you can apply not only background environmental image but also a HDR image for image based lighting (IBL). For applying environmental image use skybox(:_) type method.

    func skybox(_ resource: EnvironmentResource) -> ARView.Environment.Background

    This static method uses EnvironmentResource object that contains background data for your scene. You can read about it here and in my story on Medium.

    Also, in RealityKit for iOS you are capable of applying a vertex/fragment shader (via CustomMaterial) and get the rendered scene as a texture (like you could do in SceneKit+Metal).

    To find out how to disable a default environment lighting in ARView read this post.


    visionOS RealityKit

    Use RealityRenderer object to get access to customization of environment image and IBL:

    RealityRenderer().lighting.resource = try! .load(named: "env")

    This post, shows you how to enable a custom environment lighting in visionOS app.