Search code examples
cimageimage-processingvideoyuv

Processing YUV I420 from framebuffer?

I have a byte array named buf, that contains a single video frame in YUV I420 format obtained from a framebuffer. For every video frame I also have the following information:

Size (e.g. 320x180)
Stride Y (e.g. 384)
Stride U (e.g. 384)
Stride V (e.g. 384)
Plane offset Y (e.g. 0)
Plane offset U (e.g. 69120)
Plane offset V (e.g. 69312)

Concatenating multiple video frames in a file, and passing that with size information to a raw video decoder in VLC or FFmpeg just produces garbled colors, so I think the bytes in buf should be reordered using the information above to produce playable output, but I'm completely new to working with video so this may be wrong.

I which order should size, stride and offset information be combined with bytes in buf to produce a byte stream that could be played raw in a video player?

Example:

https://transfer.sh/E8LNy5/69518644-example-01.yuv

Solution

  • The layout of the data seems odd but using the given offsets and strides, this is decodable as YUV.

    First there are 384 * 180 bytes of luma.

    Following are the chroma lines, each being 192 bytes long... but U and V lines take turns! This is accounted for by the strange offsets. U offset points exactly to after luma. V offset is 192 bytes further... and reading would leapfrog by 384 bytes.

    Here's code that extracts those planes and assembles them as I420, for decoding with cvtColor:

    #!/usr/bin/env python3

import numpy as np
import cv2 as cv

def extract(data, offset, stride, width, height):
    data = data[offset:] # skip to...
    data = data[:height * stride] # get `height` lines
    data.shape = (height, stride)
    return data[:, :width] # drop overscan/padding

width, height = 320, 180

Yoffset = 0
Uoffset = 69120 # 384*180
Voffset = 69312 # 384*180 + 192

Ystride = 384
Ustride = 384
Vstride = 384

data = np.fromfile("69518644-example-01.yuv", dtype=np.uint8)

Y = extract(data, Yoffset, Ystride, width, height)
U = extract(data, Uoffset, Ustride, width // 2, height // 2)
V = extract(data, Voffset, Vstride, width // 2, height // 2)

# construct I420: Y,U,V planes in order

i420 = np.concatenate([Y.flat, U.flat, V.flat])
i420.shape = (height * 3 // 2, width)

result = cv.cvtColor(i420, cv.COLOR_YUV2BGR_I420)

cv.namedWindow("result", cv.WINDOW_NORMAL)
cv.resizeWindow("result", width * 4, height * 4)
cv.imshow("result", result)
cv.waitKey()
cv.destroyAllWindows()

    output