Search code examples
gstreamertegravideo-conversionopenmaxvp9

Gstreamer videoconvert color conversion wrong?

I'm launching a gst-launch-1.0 that captures camera images with nvgstcamera. The images are encoded to VP9 video. The video is tee'd to a filesink that saves the video in a webm container and to a VP9 decoder that pipes the images into an appsink.

Later, I want to extract frames from the saved video and run them through the application again. It is important that the frames are absolutely identical to the ones that were piped into the appsink during video capture.

Unfortunately, the decoded frames look slightly different, depending on how you extract them.

A minimal working example:

Recording:

$ gst-launch-1.0 nvcamerasrc ! "video/x-raw(memory:NVMM), format=NV12" ! omxvp9enc ! tee name=splitter \
  splitter. ! queue ! webmmux ! filesink location="record.webm" \
  splitter. ! queue ! omxvp9dec ! nvvidconv ! "video/x-raw,format=RGBA" ! pngenc ! multifilesink location="direct_%d.png"

Replaying with nvvidconv element:

$ gst-launch-1.0 filesrc location=record.webm ! matroskademux ! omxvp9dec \
! nvvidconv ! pngenc ! multifilesink location="extracted_nvvidconv_%d.png"

Replaying with videoconvert element:

$ gst-launch-1.0 filesrc location=record.webm ! matroskademux ! omxvp9dec \
! videoconvert ! pngenc ! multifilesink location="extracted_videoconvert_%d.png"

Testing image differences:

$ compare -metric rmse direct_25.png extracted_nvvidconv_25.png null
0

$ compare -metric rmse direct_25.png extracted_videoconvert_25.png null
688.634 (0.0105079)

nvvidconv: nvvidconv

videoconvert: videoconvert

My guess is that this has to do with the I420 to RGB conversion. So videoconvert seems to use a different color conversion than nvvidconv.

Launching the pipeline with gst-launch -v shows that the element capabilities are basically the same for both replay pipelines, the only difference is that videoconvert uses RGB by default, while nvvidconv uses RGBA. Adding the caps string "video/x-raw,format=RGBA" behind videoconvert makes however no difference in color conversion.

Note that this is on an Nvidia Jetson TX2 and I would like to use hardware accelerated gstreamer plugins during recording (omxvp9enc, nvvidconv), but not during replay on another machine.

How can I extract images from the video that are identical to the images running through the pipeline during recording, but without the use of Nvidia's Jetson-specific plugins?

Solution

  • Check for colorimetry information - https://developer.gnome.org/gst-plugins-libs/stable/gst-plugins-base-libs-gstvideo.html#GstVideoColorimetry

    Videoconvert for example take these into account when converting images. Depending on the caps found at input and output.

    You probably have to check what the Tegra is doing here. Most likely there is a difference if the signal is interpreted as full range or tv range. Or the matrices differ from 601 and 709.

    Depending on precision there may still be some loss during the conversion. For metrics at video codecs it may make sense to stay at YUV color space and use only RGB for display if you must.