Search code examples
iosimage-processinggpuimage

GPUImage : YUV or RGBA impact on performance?

I'm working on some still image processing, and GPUImage is a really awesome framework (thank you Brad Larson!).

I understand that :

  • some filters can be done with only 1 component. In this case, the image should be YUV (YCbCr), and we use only Y (luma = image grey level).
  • other filters need all color information from the 3 components - R,G and B.
  • YUV -> RGB conversion is provided (in GPUVideoCamera), RGB -> YUV may be hard-coded into the fragment shader (ex: GPUImageChromaKeyFilter)

I have many image-processing steps, some which can be based on YUV, others on RGB. Basically, I want to mix RGB and YUV filters, so my general question is this :

What is the cost / information-loss of such successive conversions, and would you recommend any design ?

Thanks!

(PS : what is the problem with iPhone4 YUV->RGB conversion & AVCaptureStillImageOutput pixel-Format ?)

Solution

  • The use of YUV in GPUImage is a fairly new addition, and something I'm still experimenting with. I wanted to pull in YUV to try to improve filter performance, reduce memory usage, and possibly increase color fidelity. So far, my modifications have only achieved one of these three.

    As you can see, I pull in YUV frames from the camera and then decide what to do with them at subsequent stages in the filter pipeline. If all of the filters that the camera input targets only want monochrome inputs, the camera input will send only the unprocessed Y channel texture on down the pipeline. If any of the filters need RGB input, the camera input will perform a shader-based conversion from YUV->RGB.

    For filters that take in monochrome, this can lead to a significant performance boost with the elimination of the RGB conversion phase (done by AV Foundation when requesting BGRA data, or in my conversion shader), as well as a redundant conversion of RGB back to luminance. On an iPhone 4, the performance of the Sobel edge detection filter running on 720p frames goes from 36.0 ms per frame with RGB input to 15.1 ms using the direct Y channel. This also avoids a slight loss of information due to rounding from converting YUV to RGB and back to luminance. 8-bit color channels only have so much dynamic range.

    Even when using RGB inputs, the movement of this conversion out of AV Foundation and into my shader leads to a performance win. On an iPhone 4S, running a saturation filter against 1080p inputs drops from 2.2 ms per frame to 1.5 ms per frame with my conversion shader instead of AV Foundation's built-in BGRA output.

    Memory consumption is nearly identical between the two RGB approaches, so I'm experimenting with a way to improve this. For monochrome inputs, memory usage drops significantly due to the smaller texture size of the inputs.

    Implementing an all-YUV pipeline is a little more challenging, because you would need to maintain parallel rendering pathways and shaders for the Y and UV planes, with separate input and output textures for both. Extracting planar YUV from RGB is tricky, because you'd need to somehow pull two outputs from one input, something that isn't natively supported in OpenGL ES. You'd need to do two render passes, which is fairly wasteful. Interleaved YUV444 might be more practical as a color format for a multistage pipeline, but I haven't played around with this yet.

    Again, I'm just beginning to tinker with this.