CAMetalLayer leaks memory during resize

Question

I am creating a window on macOS whose entire content is a subclass of CAMetalLayer. I re-render at the display’s refresh rate using a CVDisplayLink; the window is constantly being re-rendered, even at idle.

Each time I resize the window, memory usage increases by hundreds of megabytes per second.

While the window is not being resized, memory usage increases by maybe 0.3 MB/s (a small memory leak because I haven’t included an @autoreleasepool anywhere – this is ok for demo purposes). This makes me think that new framebuffers are being allocated during resize, but are never freed.

I’ve investigated using Instruments.app, and all this memory is being allocated in CAMetalLayer’s nextDrawable method.

Here is the code most significant to the question:

// ...

- (void)setFrameSize:(NSSize)size
{
    [super setFrameSize:size];
    metalLayer.drawableSize = size;
}

// ...

- (void)renderOneFrame
{
    id<CAMetalDrawable> drawable = [metalLayer nextDrawable];
    id<MTLCommandBuffer> commandBuffer = [commandQueue commandBuffer];

    MTLRenderPassDescriptor* passDesc = [[MTLRenderPassDescriptor alloc] init];
    passDesc.colorAttachments[0].texture = drawable.texture;
    passDesc.colorAttachments[0].loadAction = MTLLoadActionClear;
    passDesc.colorAttachments[0].storeAction = MTLStoreActionStore;

    id<MTLRenderCommandEncoder> commandEncoder =
            [commandBuffer renderCommandEncoderWithDescriptor:passDesc];
    [commandEncoder setRenderPipelineState:renderPipeline];

    [commandEncoder endEncoding];

    [commandBuffer presentDrawable:drawable];
    [commandBuffer commit];
}

// ...

This really is a minimal working example – I’m not even rendering anything!

Note how I ask Metal to resize the frame buffer simply by modifying CAMetalLayer’s drawableSize property. I checked the docs both for that and for nextDrawable, but I didn’t see anything that would be relevant to this.

Apple’s example code for CAMetalLayer resizes the layer in the same way I do, and I couldn’t see it doing anything else that might affect this (though clearly I’m wrong since the Apple example code doesn’t leak memory like my code does).

---

Just to be safe, here is my full source code:

// main.m

#import <Cocoa/Cocoa.h>
#import <Metal/Metal.h>
#import <QuartzCore/QuartzCore.h>

@interface MainView : NSView {
    CAMetalLayer* metalLayer;
    CVDisplayLinkRef displayLink;
    id<MTLDevice> device;
    id<MTLCommandQueue> commandQueue;
    id<MTLRenderPipelineState> renderPipeline;
}
@end

@implementation MainView
- (id)initWithFrame:(CGRect)frame
{
    self = [super initWithFrame:frame];
    self.wantsLayer = YES;
    self.layer = [CAMetalLayer layer];
    metalLayer = (CAMetalLayer*)self.layer;
    device = MTLCreateSystemDefaultDevice();
    metalLayer.device = device;

    NSURL* path = [NSURL fileURLWithPath:@"out/shaders.metallib" isDirectory:false];
    id<MTLLibrary> library = [device newLibraryWithURL:path error:nil];
    MTLRenderPipelineDescriptor* desc = [[MTLRenderPipelineDescriptor alloc] init];
    desc.vertexFunction = [library newFunctionWithName:@"vertexShader"];
    desc.fragmentFunction = [library newFunctionWithName:@"fragmentShader"];
    renderPipeline = [device newRenderPipelineStateWithDescriptor:desc error:nil];
    commandQueue = [device newCommandQueue];

    CVDisplayLinkCreateWithActiveCGDisplays(&displayLink);
    CVDisplayLinkSetOutputCallback(displayLink, displayLinkCallback, self);
    CVDisplayLinkStart(displayLink);

    return self;
}

- (void)setFrameSize:(NSSize)size
{
    [super setFrameSize:size];
    metalLayer.drawableSize = size;
}

static CVReturn displayLinkCallback(
        CVDisplayLinkRef displayLink,
        const CVTimeStamp* now,
        const CVTimeStamp* outputTime,
        CVOptionFlags flagsIn,
        CVOptionFlags* flagsOut,
        void* displayLinkContext)
{
    [(MainView*)displayLinkContext renderOneFrame];
    return kCVReturnSuccess;
}

- (void)renderOneFrame
{
    id<CAMetalDrawable> drawable = [metalLayer nextDrawable];
    id<MTLCommandBuffer> commandBuffer = [commandQueue commandBuffer];

    MTLRenderPassDescriptor* passDesc = [[MTLRenderPassDescriptor alloc] init];
    passDesc.colorAttachments[0].texture = drawable.texture;
    passDesc.colorAttachments[0].loadAction = MTLLoadActionClear;
    passDesc.colorAttachments[0].storeAction = MTLStoreActionStore;

    id<MTLRenderCommandEncoder> commandEncoder =
            [commandBuffer renderCommandEncoderWithDescriptor:passDesc];
    [commandEncoder setRenderPipelineState:renderPipeline];

    [commandEncoder endEncoding];

    [commandBuffer presentDrawable:drawable];
    [commandBuffer commit];
}
@end

int main()
{
    [NSApplication sharedApplication];
    [NSApp setActivationPolicy:NSApplicationActivationPolicyRegular];

    NSRect rect = NSMakeRect(100, 100, 500, 400);

    NSWindow* window = [NSWindow alloc];
    [window initWithContentRect:rect
                      styleMask:NSWindowStyleMaskTitled | NSWindowStyleMaskResizable
                        backing:NSBackingStoreBuffered
                          defer:NO];
    MainView* view = [[MainView alloc] initWithFrame:rect];
    window.contentView = view;

    [window makeKeyAndOrderFront:nil];

    [NSApp activateIgnoringOtherApps:YES];
    [NSApp run];
}

And here are my shaders:

// shaders.metal

struct V {
    float4 p [[position]];
};

vertex V vertexShader(const device V* v [[buffer(0)]], uint i [[vertex_id]])
{
    return v[i];
}

fragment float4 fragmentShader(V v [[stage_in]])
{
    return v.p;
}

Shaders can be compiled with xcrun -sdk macosx metal shaders.metal -o out/shaders.metallib, and the Objective C source can be compiled with clang -framework Cocoa -framework QuartzCore -framework Metal -o out/example main.m.

Answer 1

From apple sample code that you provide:

The CVDisplayLink callback, displayLinkCallback, never executes
on the main thread.

So you need to add autoreleasepool to your callback function:

static CVReturn displayLinkCallback(
        CVDisplayLinkRef displayLink,
        const CVTimeStamp* now,
        const CVTimeStamp* outputTime,
        CVOptionFlags flagsIn,
        CVOptionFlags* flagsOut,
        void* displayLinkContext)
{
    @autoreleasepool {
    [(MainView*)displayLinkContext renderOneFrame];
    }

    return kCVReturnSuccess;
}

It looks like ARC is disabled in your project. To explicitly enable ARC add the -fobjc-arc flag to your compile command.

clang -fobjc-arc -framework Cocoa -framework QuartzCore -framework Metal -o out/example main.m