Fastest way to render OpenGL texture into CGContext

Question

Here's the question in brief:

For some layer compositing, I have to render an OpenGL texture in a CGContext. What's the fastest way to do that?

Thoughts so far:

Obviously, calling renderInContext won't capture OpenGL content, and glReadPixels is too slow.

For some 'context', I'm calling this method in a delegate class of a layer:

- (void) drawLayer:(CALayer *)layer inContext:(CGContextRef)ctx

I've considered using a CVOpenGLESTextureCache, but that requires an additional rendering, and it seems like some complicated conversion would be necessary post-rendering.

Here's my (terrible) implemention right now:

glBindRenderbuffer(GL_RENDERBUFFER, displayRenderbuffer);

NSInteger x = 0, y = 0, width = backingWidth, height = backingHeight;
NSInteger dataLength = width * height * 4;
GLubyte *data = (GLubyte *) malloc(dataLength * sizeof(GLubyte));

glPixelStorei(GL_PACK_ALIGNMENT, 4);
glReadPixels(x, y, width, height, GL_RGBA, GL_UNSIGNED_BYTE, data);

CGDataProviderRef ref = CGDataProviderCreateWithData(NULL, data, dataLength, NULL);
CGColorSpaceRef colorspace = CGColorSpaceCreateDeviceRGB();
CGImageRef iref = CGImageCreate(width, height, 8, 32, width * 4, colorspace, kCGBitmapByteOrder32Big | kCGImageAlphaPremultipliedLast,
                                ref, NULL, true, kCGRenderingIntentDefault);

CGFloat scale = self.contentScaleFactor;
NSInteger widthInPoints, heightInPoints;
widthInPoints = width / scale;
heightInPoints = height / scale;

CGContextSetBlendMode(context, kCGBlendModeCopy);
CGContextDrawImage(context, CGRectMake(0.0, 0.0, widthInPoints, heightInPoints), iref);

// Clean up
free(data);
CFRelease(ref);
CFRelease(colorspace);
CGImageRelease(iref);

Answer 1

For anyone curious, the method shown above is not the fastest way.

When a UIView is asked for its contents, it will ask its layer (usually a CALayer) to draw them for it. The exception: OpenGL-based views, which use a CAEAGLLayer (a subclass of CALayer), use the same method but returns nothing. No drawing happens.

So, if you call:

[someUIView.layer drawInContext:someContext];

it will work, while

[someOpenGLView.layer drawInContext:someContext];

won't.

This also becomes an issue if you're asking a superview of any OpenGL-based view for its content: it will recursively ask each of its subviews for theirs, and any subview that uses a CAEAGLLayer will hand back nothing (you'll see a black rectangle).

I set out above to find an implementation of a delegate method of CALayer, drawLayer:inContext:, which I could use in any OpenGL-based views so that the view object itself would provide its contents (rather than the layer). The delegate method is called automatically: Apple expects it to work this way.

Where performance isn't an issue, you can implement a variation of a simple snapshot method in your view. The method would look like this:

- (void) drawLayer:(CALayer *)layer inContext:(CGContextRef)ctx {

    GLint backingWidth, backingHeight;

    glBindRenderbufferOES(GL_RENDERBUFFER_OES, _colorRenderbuffer);

    glGetRenderbufferParameterivOES(GL_RENDERBUFFER_OES, GL_RENDERBUFFER_WIDTH_OES, &backingWidth);
    glGetRenderbufferParameterivOES(GL_RENDERBUFFER_OES, GL_RENDERBUFFER_HEIGHT_OES, &backingHeight);

    NSInteger x = 0, y = 0, width = backingWidth, height = backingHeight;
    NSInteger dataLength = width * height * 4;
    GLubyte *data = (GLubyte*)malloc(dataLength * sizeof(GLubyte));

    // Read pixel data from the framebuffer
    glPixelStorei(GL_PACK_ALIGNMENT, 4);
    glReadPixels(x, y, width, height, GL_RGBA, GL_UNSIGNED_BYTE, data);

    CGDataProviderRef ref = CGDataProviderCreateWithData(NULL, data, dataLength, NULL);
    CGColorSpaceRef colorspace = CGColorSpaceCreateDeviceRGB();
    CGImageRef iref = CGImageCreate(width, height, 8, 32, width * 4, colorspace, kCGBitmapByteOrder32Big | kCGImageAlphaPremultipliedLast,
                                ref, NULL, true, kCGRenderingIntentDefault);
    CGContextDrawImage(ctx, self.bounds, iref);
}

BUT! This is not a performance effective.

glReadPixels, as noted just about everywhere, is not a fast call. Starting in iOS 5, Apple exposed CVOpenGLESTextureCacheRef - basically, a shared buffer that can be used both as a CVPixelBufferRef and as an OpenGL texture. Originally, it was designed to be used as a way of getting an OpenGL texture from a video frame: now it's more often used in reverse, to get a video frame from a texture.

So a much better implementation of the above idea is to use the CVPixelBufferRef you get from CVOpenGLESTextureCacheCreateTextureFromImage, get direct access to those pixels, draw them into a CGImage which you cache and which is drawn into your context in the delegate method above.

The code is here. On each rendering pass, you draw your texture into the texturecache, which is linked to the CVPixelBuffer Ref:

- (void) renderToCGImage {

 // Setup the drawing
[ochrContext useProcessingContext];
glBindFramebuffer(GL_FRAMEBUFFER, layerRenderingFramebuffer);
glViewport(0, 0, (int) self.frame.size.width, (int) self.frame.size.height);
[ochrContext setActiveShaderProgram:layerRenderingShaderProgram];

// Do the actual drawing
glActiveTexture(GL_TEXTURE4);
glBindTexture(GL_TEXTURE_2D, self.inputTexture);
glUniform1i(layerRenderingInputTextureUniform, 4);
glVertexAttribPointer(layerRenderingShaderPositionAttribute, 2, GL_FLOAT, 0, 0, kRenderTargetVertices);
glVertexAttribPointer(layerRenderingShaderTextureCoordinateAttribute, 2, GL_FLOAT, 0, 0, kRenderTextureVertices);

// Draw and finish up
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
glFinish();

// Try running this code asynchronously to improve performance
dispatch_async(PixelBufferReadingQueue, ^{
    
    // Lock the base address (can't get the address without locking it).
    CVPixelBufferLockBaseAddress(renderTarget, 0);
    
    // Get a pointer to the pixels
    uint32_t * pixels = (uint32_t*) CVPixelBufferGetBaseAddress(renderTarget);
    
    // Wrap the pixel data in a data-provider object.
    CGDataProviderRef pixelWrapper = CGDataProviderCreateWithData(NULL, pixels, CVPixelBufferGetDataSize(renderTarget), NULL);
    
    // Get a color-space ref... can't this be done only once?
    CGColorSpaceRef colorSpaceRef = CGColorSpaceCreateDeviceRGB();
    
    // Release the exusting CGImage
    CGImageRelease(currentCGImage);
    
    // Get a CGImage from the data (the CGImage is used in the drawLayer: delegate method above)
    currentCGImage = CGImageCreate(self.frame.size.width,
                                   self.frame.size.height,
                                   8,
                                   32,
                                   4 * self.frame.size.width,
                                   colorSpaceRef,
                                   kCGBitmapByteOrder32Big | kCGImageAlphaPremultipliedLast,
                                   pixelWrapper,
                                   NULL,
                                   NO,
                                   kCGRenderingIntentDefault);
    
    // Clean up
    CVPixelBufferUnlockBaseAddress(renderTarget, 0);
    CGDataProviderRelease(pixelWrapper);
    CGColorSpaceRelease(colorSpaceRef);
    
    });
}

And then implement the delegate method very simply:

- (void) drawLayer:(CALayer *)layer inContext:(CGContextRef)ctx {
CGContextDrawImage(ctx, self.bounds, currentCGImage);
}