Getting RGBA values for all pixels of CGImage Swift

Question

I am trying to create a real time video processing app, in which I need to get the RGBA values of all pixels for each frame, and process them using an external library, and show them. I am trying to get the RGBA value for each pixel, but it is too slow the way I am doing it, I was wondering if there is a way to do it faster, using VImage. This is my current code, and the way I get all the pixels, as I get the current frame:

        guard let cgImage = context.makeImage() else {
        return nil
    }
    guard let data = cgImage.dataProvider?.data,
    let bytes = CFDataGetBytePtr(data) else {
    fatalError("Couldn't access image data")
    }
    assert(cgImage.colorSpace?.model == .rgb)
    let bytesPerPixel = cgImage.bitsPerPixel / cgImage.bitsPerComponent
    gp.async {
        for y in 0 ..< cgImage.height {
            for x in 0 ..< cgImage.width {
                let offset = (y * cgImage.bytesPerRow) + (x * bytesPerPixel)
                let components = (r: bytes[offset], g: bytes[offset + 1], b: bytes[offset + 2])
                print("[x:\(x), y:\(y)] \(components)")
            }
            print("---")
        }

    }

This is the version using the VImage, but I there is some memory leak, and I can not access the pixels

        guard
        let format = vImage_CGImageFormat(cgImage: cgImage),
        var buffer = try? vImage_Buffer(cgImage: cgImage,
                                        format: format) else {
            exit(-1)
        }

    let rowStride = buffer.rowBytes / MemoryLayout<Pixel_8>.stride / format.componentCount
    do {
        
        let componentCount = format.componentCount
        var argbSourcePlanarBuffers: [vImage_Buffer] = (0 ..< componentCount).map { _ in
            guard let buffer1 = try? vImage_Buffer(width: Int(buffer.width),
                                                   height: Int(buffer.height),
                                                  bitsPerPixel: format.bitsPerComponent) else {
                                                    fatalError("Error creating source buffers.")
            }
            return buffer1
        }
        vImageConvert_ARGB8888toPlanar8(&buffer,
                                        &argbSourcePlanarBuffers[0],
                                        &argbSourcePlanarBuffers[1],
                                        &argbSourcePlanarBuffers[2],
                                        &argbSourcePlanarBuffers[3],
                                        vImage_Flags(kvImageNoFlags))

        let n = rowStride * Int(argbSourcePlanarBuffers[1].height) * format.componentCount
        let start = buffer.data.assumingMemoryBound(to: Pixel_8.self)
        var ptr = UnsafeBufferPointer(start: start, count: n)

        print(Array(argbSourcePlanarBuffers)[1]) // prints the first 15 interleaved values
        buffer.free()
    }

Answer 1

You can access the underlying pixels in a vImage buffer to do this.

For example, given an image named cgImage, use the following code to populate a vImage buffer:

guard
    let format = vImage_CGImageFormat(cgImage: cgImage),
    let buffer = try? vImage_Buffer(cgImage: cgImage,
                                    format: format) else {
        exit(-1)
    }

let rowStride = buffer.rowBytes / MemoryLayout<Pixel_8>.stride / format.componentCount

Note that a vImage buffer's data may be wider than the image (see: https://developer.apple.com/documentation/accelerate/finding_the_sharpest_image_in_a_sequence_of_captured_images) which is why I've added rowStride.

To access the pixels as a single buffer of interleaved values, use:

do {
    let n = rowStride * Int(buffer.height) * format.componentCount
    let start = buffer.data.assumingMemoryBound(to: Pixel_8.self)
    let ptr = UnsafeBufferPointer(start: start, count: n)
    
    print(Array(ptr)[ 0 ... 15]) // prints the first 15 interleaved values
}

To access the pixels as a buffer of Pixel_8888 values, use (make sure that format.componentCount is 4:

do {
    let n = rowStride * Int(buffer.height)
    let start = buffer.data.assumingMemoryBound(to: Pixel_8888.self)
    let ptr = UnsafeBufferPointer(start: start, count: n)
    
    print(Array(ptr)[ 0 ... 3]) // prints the first 4 pixels
}