I'm new to Metal. I'm rendering a SceneKit scene with Metal using this Apple sample code. TLDR; it calls the SCNRenderer's render function and passes in a command buffer. I'm compiling for Big Sur.
It works, but it is not anti-aliased. I've tried a few ways to achieve it, as you can see in the updates below.
Without Metal, I'd just set isJitteringEnabled to true on the SCNRenderer, and I get beautiful (and slow) 96-ish-pass renderings. If I try to do this with Metal, I get weird pixel format mismatches, so I'm suspecting the two just aren't compatible.
With Metal, as far as I can tell, the simplest way to achieve antialiasing is to enable multi-sampling in the render pipeline (I know how to do that) — and use a multi sampling texture (MTLTextureType.type2DMultisample). This partial answer backs up my assumption.
And that's the problem. I don't know how to change the texture type when I get my texture from CVMetalTextureCache and CVMetalTextureCacheCreateTextureFromImage. It seems this is a limitation in Core Video's Metal support?
My full source is here
That's it. The rest of this post is more details on the stuff I tried.
(I think this might be possible using a shader. I'm open to that solution as well, but I don't know where to start. This example doesn't compile, and this example is for GSLS)
My pixel buffer atts look like this
let pixelbufferAttributes = [
kCVPixelBufferPixelFormatTypeKey : kCVPixelFormatType_32BGRA,
kCVPixelBufferWidthKey: exportSettings.width,
kCVPixelBufferHeightKey : exportSettings.height,
kCVPixelBufferMetalCompatibilityKey: true] as [String: Any]
For each frame, it creates a new pixel buffer from the pool, wraps it in a Metal texture from a cache, like this
let pixelFormat = MTLPixelFormat.bgra8Unorm_srgb
var optionalMetalTexture: CVMetalTexture?
err = CVMetalTextureCacheCreateTextureFromImage(
kCFAllocatorDefault,
metalTextureCache, // object prop
pixelBuffer,
nil, // texture attributes
pixelFormat,
exportSettings.width,
exportSettings.height,
0, // planeIndex
&optionalMetalTexture)
guard err == noErr, let metalTexture = optionalMetalTexture else {
fatalError("Failed to create metal texture wrapper from pixel bufffer \(err)")
}
Attempt: Change the texture descriptor
Since I'm creating my Metal texture from a CVPixelbuffer with CVMetalTextureCacheCreateTextureFromImage, I can't figure out how to set its attributes and make it multi sample.
Attempt: Try H264
Didn't change anything. Also tried changing just the alpha quality, with HEVC with alpha, but no change.
Attempt: Enable multi sampling
I was able to get my pipeline to pick up that I wanted multi sampling, but it crashes due to the texture not being set up for multisampling (more precisely a MTLTexture of type .2DMultisample (docs)
Attempt: Copy the MTLTexture created by Core Video
I tried to use a MTLBlitCommandEncoder to copy the texture I was given by Core Video into a texture I had set up with the right attributes. But it crashes telling me that the attributes don't match.
I'm starting to think there's no solution to this?
Enabling multisampling was the right idea. The following patch shows how to enable it.
--- a/HEVC-Videos-With-Alpha-AssetWriting/HEVC-Videos-With-Alpha-AssetWriting/AppDelegate.swift
+++ b/HEVC-Videos-With-Alpha-AssetWriting/HEVC-Videos-With-Alpha-AssetWriting/AppDelegate.swift
@@ -32,6 +32,8 @@ class AppDelegate: NSObject, NSApplicationDelegate, SCNSceneRendererDelegate {
let renderer = SCNRenderer(device: nil, options: nil)
var lampMaterials: SCNNode!
var metalTextureCache: CVMetalTextureCache!
+ let msaaSampleCount = 1
+ var metalMultisampledTexture: MTLTexture!
// Export
var frameCounter = 0
@@ -61,6 +63,18 @@ class AppDelegate: NSObject, NSApplicationDelegate, SCNSceneRendererDelegate {
fatalError("Cannot create metal texture cache: \(err)")
}
metalTextureCache = optionalMetalTextureCache
+
+ if (msaaSampleCount > 1) {
+ let textureDescriptor = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: MTLPixelFormat.bgra8Unorm_srgb,
+ width: ExportSettings.width,
+ height: ExportSettings.height,
+ mipmapped: false)
+ textureDescriptor.usage = .renderTarget
+ textureDescriptor.storageMode = .private
+ textureDescriptor.textureType = .type2DMultisample
+ textureDescriptor.sampleCount = msaaSampleCount
+ metalMultisampledTexture = renderer.device!.makeTexture(descriptor: textureDescriptor)
+ }
}
/// Render next frame and call the frame completion handler
@@ -106,7 +120,14 @@ class AppDelegate: NSObject, NSApplicationDelegate, SCNSceneRendererDelegate {
let renderPassDescriptor = MTLRenderPassDescriptor()
renderPassDescriptor.colorAttachments[0].loadAction = .clear
renderPassDescriptor.colorAttachments[0].clearColor = clearColor
- renderPassDescriptor.colorAttachments[0].texture = CVMetalTextureGetTexture(metalTexture)
+ if (msaaSampleCount > 1) {
+ renderPassDescriptor.colorAttachments[0].texture = metalMultisampledTexture
+ renderPassDescriptor.colorAttachments[0].resolveTexture = CVMetalTextureGetTexture(metalTexture)
+ renderPassDescriptor.colorAttachments[0].storeAction = .multisampleResolve
+ }
+ else {
+ renderPassDescriptor.colorAttachments[0].texture = CVMetalTextureGetTexture(metalTexture)
+ }
renderer.render(atTime: currentPresentationTime.seconds,
viewport: ExportSettings.viewport,
commandBuffer: commandBuffer,