Search code examples
iosavfoundationavplayeritemcore-videocvpixelbuffer

AVPlayerItemVideoOutput copyPixelBufferForItemTime gives incorrect CVPixelBufferRef on iOS for particular video

Have you met the problem when for the same video copyPixelBufferForItemTime is incorrect on iOS?

I have AVPlayerItemVideoOutput, linked to appropriate AVPlayerItem. I call copyPixelBufferForItemTime, receive CVPixelBufferRef and then retrieve OpenGL texture from it.

CVPixelBufferRef pb = [_playerVideoOutput copyPixelBufferForItemTime:currentTime itemTimeForDisplay:nil];

For this sample video there's bug with CVPixelBufferRef:

int bpr = (int)CVPixelBufferGetBytesPerRow(pb);
int width_real = (int)CVPixelBufferGetWidth(pb);
int width_working = (int)CVPixelBufferGetBytesPerRow(pb)/4;

Mac output:
bpr = 2400
width_real = 596
width_working = 600

iOS output:
bpr = 2432
width_real = 596
width_working = 608

How it's rendered on iOS:
enter image description here

How it's rendered on Mac:
enter image description here

CVPixelBufferGetPixelFormatType returns BGRA on both platforms.

Edit When creating texture on iOS, I read data from pixel buffer via CVPixelBufferGetBaseAddress and use provided size CVPixelBufferGetWidth/CVPixelBufferGetHeight: 

- (GLuint)createTextureFromMovieFrame:(CVPixelBufferRef)movieFrame
{
    int bufferWidth = (int) CVPixelBufferGetWidth(movieFrame);
    int bufferHeight = (int) CVPixelBufferGetHeight(movieFrame);

    // Upload to texture
    CVPixelBufferLockBaseAddress(movieFrame, 0);

    CVOpenGLTextureRef texture=0;
    GLuint tex = 0;

#if TARGET_OS_IOS==1
    void * data = CVPixelBufferGetBaseAddress(movieFrame);
    CVReturn err = 0;
    tex = algotest::MyGL::createRGBATexture(bufferWidth, bufferHeight, data, algotest::MyGL::KLinear);

#else
    CVReturn err = CVOpenGLTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
                                                          getGlobalTextureCache(), movieFrame, 0, &texture);
#endif

    CVPixelBufferUnlockBaseAddress(movieFrame, 0);

    return tex;
}

So width_working is just for debug. As it mismatch width_real, and passing neither width_working not width_real doesn't work, I suppose that it's a bug with pixel buffer.

Solution

  • The pixel buffers have per-line padding pixels on both iOS and mac, presumably for alignment reasons. The difference is that the mac CVOpenGLTextureCacheCreateTextureFromImage function understands this, while the iOS createRGBATexture function can not, not without a bytes-per-row argument.

    You could either include the padding pixels in the width, and crop them out later:

    tex = algotest::MyGL::createRGBATexture(CVPixelBufferGetBytesPerRow(movieFrame)/4, bufferHeight, data, algotest::MyGL::KLinear);

    Or you could use CVOpenGLESTextureCache, the iOS equivalent of CVOpenGLTextureCache and replace createRGBATexture() with CVOpenGLESTextureCacheCreateTextureFromImage(). Then your mac & iOS code would be similar & the iOS code might even run faster as texture caches on iOS can avoid redundant copying of texture data.