I want to create a library that can convert any video frame captured using AVCaptureDevice to target pixel format. So, it must support kCVPixelFormatType_32BGRA, kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, kCVPixelFormatType_420YpCbCr8BiPlanarFullRange and optionally kCVPixelFormatType_420YpCbCr8Planar.
For performance I want to use Accelerate framework as it contains rich set of conversion functions. As target pixel format may be different and is set by library user it'll be grate to use universal vImageConvert_AnyToAny function:
You use vImage's vImageConvert_AnyToAny(::::_:) function to convert between Core Video or Core Graphics image data of arbitrary color spaces and bit depths. The source and destination images are described by one or more buffers. For example, a Y'CbCr image may be composed of one buffer containing luminance information and one buffer containing chrominance information.
To use this function I must create vImageConverter which define conversion between images. Constructors of this class require source and destination image format description in form of vImage_CGImageFormat:
vImage_CGImageFormat in_format = ...;
vImage_CGImageFormat out_format = ...;
vImageConverterRef converter = vImageConverter_CreateWithCGImageFormat(&in_format, &out_format, NULL, kvImagePrintDiagnosticsToConsole, &err);
if( err == kvImageNoError )
{
vImage_Buffer *src_planes = ...;
vImage_Buffer *dst_planes = ...;
err = vImageConvert_AnyToAny(converter, src_planes, dst_planes, NULL, kvImagePrintDiagnosticsToConsole);
}
The code is based on this Apple article: Building a Basic Conversion Workflow
For kCVPixelFormatType_32BGRA such vImage_CGImageFormat is simple and described in vImage_Utilities.h:
/*!
* @struct vImage_CGImageFormat
* @abstract A pixel format
* @discussion A vImage_CGImageFormat describes the ordering of the color channels, how many there are,
* the size and type of the data in the color channels and whether the data is premultiplied by alpha or not.
* This format mirrors the image format descriptors used by CoreGraphics to create things like CGImageRef and
* CGBitmapContextRef.
*
* This vImage_CGImageFormat:
*
* <pre>@textblock
* vImage_CGImageFormat format = {
* .bitsPerComponent = 8,
* .bitsPerPixel = 32,
* .colorSpace = CGColorSpaceCreateDeviceRGB(), // don't forget to release this!
* .bitmapInfo = kCGImageAlphaPremultipliedFirst | kCGBitmapByteOrder32Little,
* .version = 0, // must be 0
* .decode = NULL,
* .renderingIntent = kCGRenderingIntentDefault
* };
* @/textblock</pre>
*
* codes for a little endian ARGB8888 pixel, or what is called in the rest of vImage, BGRA8888. Note: for 16-
* and 32-bits per component formats (int16_t, uint16_t, half-float, float) most vImage image filters assume
* the data is in host-endian format. (The APIs in this header do not.) Host-endian is little endian for Intel
* and ARM, big endian for PowerPC. If the data is not in host-endian format, then you may use
* vImagePermuteChannels_ARGB8888 or vImageByteSwap_Planar16U to swap the image data byte ordering.
*
* Some examples:
* <pre>@textblock
* ARGB8888 -> {8, 32, NULL, alpha first, 0, NULL, kCGRenderingIntentDefault} alpha first = { kCGImageAlphaFirst, kCGImageAlphaPremultipliedFirst, kCGImageAlphaNoneSkipFirst }
* RGBA8888 -> {8, 32, NULL, alpha last, 0, NULL, kCGRenderingIntentDefault} alpha last = { kCGImageAlphaLast, kCGImageAlphaPremultipliedLast, kCGImageAlphaNoneSkipLast }
* BGRA8888 -> {8, 32, NULL, alpha first | kCGBitmapByteOrder32Little, 0, NULL, kCGRenderingIntentDefault}
* RGB888 -> {8, 24, NULL, kCGImageAlphaNone | kCGBitmapByteOrderDefault, 0, NULL, kCGRenderingIntentDefault}
* RGB565 -> {5, 16, NULL, kCGImageAlphaNone | kCGBitmapByteOrder16Little, 0, NULL, kCGRenderingIntentDefault}
* ARGB1555 -> {5, 16, NULL, alpha first | kCGBitmapByteOrder16Little, 0, NULL, kCGRenderingIntentDefault}
* RGBA16F -> {16, 64, NULL, alpha last | kCGBitmapFloatComponents | kCGBitmapByteOrder16Little, 0, NULL, kCGRenderingIntentDefault }
* CMYK8888 -> {8, 32, CGColorSpaceCreateDeviceCMYK(), kCGImageAlphaNone, 0, NULL, kCGRenderingIntentDefault }
* ARGBFFFF premultiplied -> {32, 128, NULL, kCGImageAlphaPremultipliedFirst | kCGBitmapFloatComponents | kCGBitmapByteOrder32Little, 0, NULL, kCGRenderingIntentDefault }
* ARGBFFFF not-premultiplied -> {32, 128, NULL, kCGImageAlphaFirst | kCGBitmapFloatComponents | kCGBitmapByteOrder32Little, 0, NULL, kCGRenderingIntentDefault }
* ARGBFFFF, alpha = 1 -> {32, 128, NULL, kCGImageAlphaNoneSkipFirst | kCGBitmapFloatComponents | kCGBitmapByteOrder32Little, 0, NULL, kCGRenderingIntentDefault }
* @/textblock</pre>
*
* Note that some of these formats, particularly RGB565 and 16F formats are supported by vImage but
* not necessarily CoreGraphics. They will be converted to a higher precision format as necessary by
* vImage in vImageCreateCGImageFromBuffer().
*
* By C rules, uninitialized struct parameters are set to zero. The last three parameters are usually zero, so can usually be omitted.
*
* <pre>@textblock
* vImage_CGImageFormat srgb888 = (vImage_CGImageFormat){
* .bitsPerComponent = 8,
* .bitsPerPixel = 24,
* .colorSpace = NULL,
* .bitmapInfo = kCGImageAlphaNone | kCGBitmapByteOrderDefault };
* @/textblock</pre>
*
* To understand how these various parameters relate to one another, we can look at the process of converting from
* one vImage_CGImageFormat format to another:
*
* 1) transform endianness of src format given by bitmapInfo to host endian (except 8 bitPerComponent content)
* 2) remove decode array transformation, and up convert to a higher range format as necessary to preserve precision / range
* 3) convert src colorspace to reference XYZ colorspace (may cause upconvert to preserve range / precision)
* 4) convert XYZ to destination colorspace + rendering intent
* 5) convert to destination precision (given by bitsPerComponent)
* 6) deal with any alpha changes (given by bitmapInfo) or flattening that needs to occur
* 7) Apply any channel reordering requested, if it didn't happen at an earlier step. (As indicated by src and dest bitmapInfo)
* 8) Apply destination decode array
* 9) Apply endianness transform given by dest bitmapInfo
*
* Clearly, for most common transformations not all steps need to occur and multiple steps can be collapsed into a compound operation.
*
* @field bitsPerComponent The number of bits needed to represent one channel of data in one pixel. For ARGB8888, this would be 8. Expected values: {1, 2, 4, 5, 8, 10, 12, 16, 32}
* @field bitsPerPixel The number of bits needed to represent one pixel. For ARGB8888, this would be 32.
* It is possible that bitsPerPixel > bitsPerComponent * number of components, but in practice this is rare.
* The number of color components is given by the colorspace and the number of alpha components (0 or 1) is given by
* by the bitmapInfo.
* @field colorSpace A description of how the pixel data in the image is positioned relative to a reference XYZ color space.
* See CoreGraphics/CGColorSpace.h. Pass NULL as a shorthand for sRGB. The vImage_CGImageFormat is not
* capable of managing the memory held by the colorSpace. If you created the colorspace, you must
* be sure to release it before all references to it disappear from scope.
* @field bitmapInfo The CGBitmapInfo describing the color channels. See CoreGraphics/CGImage.h.
* ARGB8888 is kCGImageAlphaFirst | kCGBitmapByteOrderDefault
* BGRA8888 is kCGImageAlphaFirst | kCGBitmapByteOrder32Little
* @field version The struct is versioned for future expansion. Pass 0 here.
* @field decode Prior to transformations caused by the colorspace, color channels are subject to a linear transformation.
* This allows for a different range than the typical [0,1.0]. NULL indicates default behavior of [0,1.0]
* range, and is what you should use if you don't understand this parameter. See description of CGImageCreate()
* for a discussion of decode arrays. See also Decode Arrays section of Chapter 4.8 of the PDF specification.
* The vImage_CGImageFormat is not capable of managing the memory held by the decode array. If you created a
* decode array on the heap, you must be sure to release it before all references to it disappear from scope.
*
* @field renderingIntent See CGColorSpace.h. kCGRenderingIntentDefault is typical here. By convention, rendering intent changes that
* are not accompanied by a colorspace change are ignored.
*/
I don't understand how to create vImage_CGImageFormat for YCbCr pixel format.
First I think it isn't supported at all but there are special functions for this format: Understanding YpCbCr Image Formats and vImageConverter has special functions for multi plane images such as vImageConverter_GetNumberOfSourceBuffers and vImageConverter_GetSourceBufferOrder. In the last function there are many vImage Buffer Type Codes and even kvImageBufferTypeCode_CVPixelBuffer_YCbCr.
So, it looks like it is possible to create vImageConverter for YCbCr and I need help to find out how to do it.
vImageConverter_CreateWithCGImageFormat creates a converter for converting between two Core Graphics formats.
In this case, shouldn't you be using vImageConverter_CreateForCVToCGImageFormat to convert from a CVPixelBuffer (with a vImageCVImageFormat that you can generate using vImageCVImageFormat_CreateWithCVPixelBuffer) to a CGImage? The vImage_CGImageFormat is your destination format – so the properties of that are up to you to define.