Search code examples
opencvgstreamervlc

Stream OpenCV Mat to GStreamer pipeline as RTP

I have a series of grayscale image in OpenCV, and I would like to stream it through GStreamer.
Firstly, to check that gstreamer works well, I tested it by running a simple pipeline and receiving it with VLC. (Platform is Windows 10.)

# Runs test pipeline
gst-launch-1.0 videotestsrc ! openh264enc ! h264parse ! rtph264pay config-interval=10 pt=96 ! udpsink host=127.0.0.1 port=10000

# receive.sdp
v=0
m=video 10000 RTP/AVP 96
c=IN IP4 127.0.0.1
a=rtpmap:96 H264/90000

Running receive.sdp file in VLC works fine. receive.sdp

Based on the tested pipeline, I wrote the following pipeline in which the source is cv::Mat with CV_8UC1 type.

cv::VideoWriter writer;
writer.open(
  "appsrc ! "
  "video/x-raw,format=GRAY8,width=2448,height=2048,framerate=1/1 ! "
  "videoconvert ! "
  "openh264enc ! "
  "h264parse ! "
  "rtph264pay config-interval=10 pt=96 ! "
  "udpsink host=127.0.0.1 port=10000 ",
  cv::CAP_GSTREAMER,
  0, // fourcc
  1, // fps
  cv::Size(2448, 2048), 
  false // isColor
);
// This writer is opened well, and 
// pushing cv::Mat to this writer 
// returns no error or warning code.

... and running receive.sdp in VLC, it shows nothing.

Handling grayscale in pipeline might be the problem, so I changed the pipeline to save as video file in order to check it.
The following code is saving the pipeline as a file, it works well.

cv::VideoWriter writer;
writer.open(
  "appsrc ! "
  "video/x-raw,format=GRAY8,width=2448,height=2048,framerate=1/1 ! "
  "videoconvert ! "
  "openh264enc ! "
  "h264parse ! "
  "mp4mux ! filesink location=test.mp4 ", 
  cv::CAP_GSTREAMER,
  0, // fourcc
  1, // fps
  cv::Size(2448, 2048), 
  false // isColor
);

... and following video is saved after writer.release();.
Saved video file

What am I missing?

EDIT

I found the VLC shows frame only once, after some seconds of running.
Following is log of VLC.

 main debug: VLC media player - 3.0.16 Vetinari
main debug: Copyright ⓒ 1996-2021 the VideoLAN team
main debug: revision 3.0.16-0-g5e70837d8d
main debug: configured with /home/jenkins/workspace/vlc-release/windows/vlc-release-win32-x64/extras/package/win32/../../../configure '--enable-update-check' '--enable-lua' '--enable-faad' '--enable-flac' '--enable-theora' '--enable-avcodec' '--enable-merge-ffmpeg' '--enable-dca' '--enable-mpc' '--enable-libass' '--enable-schroedinger' '--enable-realrtsp' '--enable-live555' '--enable-shout' '--enable-goom' '--enable-sse' '--enable-mmx' '--enable-libcddb' '--enable-zvbi' '--disable-telx' '--enable-nls' '--host=x86_64-w64-mingw32' '--with-contrib=../contrib/x86_64-w64-mingw32' '--with-breakpad=https://win.crashes.videolan.org' '--enable-qt' '--enable-skins2' '--enable-dvdread' '--enable-caca' 'host_alias=x86_64-w64-mingw32' 'CFLAGS= ' 'CXXFLAGS= ' 'PKG_CONFIG=pkg-config' 'PKG_CONFIG_LIBDIR=/usr/x86_64-w64-mingw32/lib/pkgconfig:/usr/lib/x86_64-w64-mingw32/pkgconfig'
h264 warning: waiting for SPS/PPS
h264 warning: waiting for SPS/PPS
h264 warning: waiting for SPS/PPS
h264 warning: waiting for SPS/PPS
h264 warning: waiting for SPS/PPS
h264 warning: waiting for SPS/PPS
h264 warning: waiting for SPS/PPS
main error: ES_OUT_SET_(GROUP_)PCR is called too late (pts_delay increased to 1851 ms)
h264 warning: waiting for SPS/PPS
main debug: ES_OUT_RESET_PCR called
main debug: Buffering 0%
main debug: Buffering 54%
h264 warning: waiting for SPS/PPS
h264 warning: waiting for SPS/PPS
main debug: Stream buffering done (3000 ms in 3000 ms)
main error: buffer deadlock prevented
main debug: Decoder wait done in 0 ms
h264 warning: waiting for SPS/PPS
h264 warning: waiting for SPS/PPS
h264 warning: waiting for SPS/PPS
h264 warning: waiting for SPS/PPS
h264 warning: waiting for SPS/PPS
h264 warning: waiting for SPS/PPS
h264 warning: waiting for SPS/PPS
h264 warning: waiting for SPS/PPS
h264 warning: waiting for SPS/PPS
h264 warning: waiting for SPS/PPS
h264 debug: found NAL_SPS (sps_id=0)
h264 debug: found NAL_PPS (pps_id=0 sps_id=0)
main debug: restarting module due to input format change
main debug: removing module "avcodec"
main debug: looking for video decoder module matching "any": 19 candidates
avcodec debug: using ffmpeg Lavc58.6.103
avcodec debug: CPU flags: 0x000fd3db
avcodec debug: allowing 6 thread(s) for decoding
avcodec debug: codec (h264) started
avcodec debug: using frame thread mode with 6 threads
main debug: using video decoder module "avcodec"
avcodec debug: available hardware decoder output format 119 (cuda)
avcodec debug: available hardware decoder output format 53 (dxva2_vld)
avcodec debug: available hardware decoder output format 118 (d3d11va_vld)
avcodec debug: available hardware decoder output format 174 (d3d11)
avcodec debug: available software decoder output format 0 (yuv420p)
avcodec debug: trying format d3d11va_vld
main debug: looking for text renderer module matching "any": 3 candidates
freetype debug: Using DWrite backend
freetype debug: DWrite_GetFamily(): family name: Arial
freetype debug: DWrite_ParseFamily(): using font at index 0 with weight 400 for bold: 0, italic: 0
freetype debug: DWrite_ParseFamily(): using font at index 1 with weight 700 for bold: 1, italic: 0
freetype debug: DWrite_ParseFamily(): using font at index 2 with weight 400 for bold: 0, italic: 1
freetype debug: DWrite_ParseFamily(): using font at index 3 with weight 700 for bold: 1, italic: 1
main debug: using text renderer module "freetype"
main debug: looking for video converter module matching "any": 24 candidates
swscale debug: 32x32 (32x32) chroma: YUVA -> 16x16 (16x16) chroma: RGBA with scaling using Bicubic (good quality)
main debug: using video converter module "swscale"
main debug: looking for video converter module matching "any": 24 candidates
yuvp debug: YUVP to YUVA converter
main debug: using video converter module "yuvp"
main debug: Deinterlacing available
main debug: deinterlace -1, mode auto, is_needed 0
main debug: looking for vout window module matching "qt,any": 3 candidates
qt debug: requesting video window...
main debug: resized to 667x466
main debug: resized to 2448x2048
main debug: using vout window module "qt"
main debug: looking for inhibit module matching "any": 1 candidates
main debug: using inhibit module "inhibit"
main debug: Opening vout display wrapper
main debug: looking for vout display module matching "any": 12 candidates
direct3d11 debug: creating Vout EventThread
direct3d11 debug: Win32VoutCreateWindow
main debug: VoutDisplayEvent 'resize' 2448x2048
qt debug: Logical video size: 2448x2048
main debug: resized to 2448x2048
main debug: VoutDisplayEvent 'resize' 2448x2048
direct3d11 debug: inclinometer not found. (hr=0x80070490)
direct3d11 debug: created video sub-window
direct3d11 debug: Vout EventThread running
direct3d11 debug: Created the D3D11 device type 1 level b100.
direct3d11 debug: different 0 device p vs P
direct3d11 warning: failed to read the 1 Display Adapter registry key (2)
direct3d11 warning: can't find adapter in registry
direct3d11 debug: supports colorspace RGB Rec.709 gamma:22 range:FULL
direct3d11 debug: supports colorspace RGB Rec.709 gamma:22 range:STUDIO
direct3d11 debug: supports colorspace RGB Rec.2020 gamma:22 range:STUDIO
direct3d11 debug: supports colorspace RGB Rec.2020 gamma:22 range:FULL
direct3d11 debug: supports colorspace RGB Rec.2020 gamma:2084 range:FULL
direct3d11 debug: supports colorspace RGB Rec.2020 gamma:2084 range:STUDIO
direct3d11 debug: Output max luminance: 270.0, colorspace RGB Rec.709 gamma:22 range:FULL, bits per pixel 8
direct3d11 debug: using colorspace RGB Rec.709 gamma:22 range:FULL
direct3d11 debug: Using pixel format VA_NV12 for chroma DX11
main debug: VoutDisplayEvent 'resize' 2470x2104
direct3d11 debug: using GPU render fence
direct3d11 debug: Direct3D11 resources created
direct3d11 debug: Direct3D11 device adapter successfully initialized
direct3d11 debug: Direct3D11 Open Succeeded
main debug: using vout display module "direct3d11"
direct3d11 debug: D3D11 pool succeed with 40 surfaces (2560x2048) context 0x0000021fe01aed70
main debug: original format sz 2448x2048, of (0,0), vsz 2448x2048, 4cc DX11, sar 1:1, msk r0x0 g0x0 b0x0
main debug: Adding 'gradient' as interactive
main debug: looking for video filter module matching "gradient,chain": 46 candidates
gradient error: Unsupported input chroma (DX11)
chain debug: Trying to use chroma I420 as middle man
main debug: looking for video converter module matching "any": 24 candidates
main debug: using video converter module "d3d11_filters"
main debug: Filter 'd3d11_filters' (0000021fdd1aee90) appended to chain
main debug: looking for video filter module matching "gradient,chain": 46 candidates
main debug: using video filter module "gradient"
main debug: Filter 'gradient' (0000021fdd1b06a0) appended to chain
main debug: using video filter module "chain"
main debug: Filter 'gradient' (0000021fdd1ad9f0) appended to chain
main debug: Adding 'sepia' as interactive
main debug: looking for video filter module matching "sepia,chain": 46 candidates
main debug: using video filter module "sepia"
main debug: Filter 'sepia' (0000021fdd1add60) appended to chain
main debug: Adding a filter to compensate for format changes
main debug: looking for video converter module matching "any": 24 candidates
d3d11_filters debug: different 0 device p vs P
d3d11_filters warning: failed to read the 1 Display Adapter registry key (2)
d3d11_filters warning: can't find adapter in registry
main debug: looking for video converter module matching "any": 24 candidates
main debug: using video converter module "i420_nv12"
main debug: using video converter module "d3d11_filters"
main debug: Filter 'd3d11_filters' (0000021fdd1af570) appended to chain
main debug: removing module "chain"
main debug: removing module "d3d11_filters"
main debug: Filter 0000021fdd1aee90 removed from chain
main debug: removing module "gradient"
main debug: Filter 0000021fdd1b06a0 removed from chain
main debug: Filter 0000021fdd1ad9f0 removed from chain
main debug: removing module "sepia"
main debug: Filter 0000021fdd1add60 removed from chain
main debug: removing module "d3d11_filters"
main debug: removing module "i420_nv12"
main debug: Filter 0000021fdd1af570 removed from chain
main debug: Adding 'gradient' as interactive
main debug: looking for video filter module matching "gradient,chain": 46 candidates
gradient error: Unsupported input chroma (DX11)
chain debug: Trying to use chroma I420 as middle man
main debug: looking for video converter module matching "any": 24 candidates
main debug: using video converter module "d3d11_filters"
main debug: Filter 'd3d11_filters' (0000021fdd1af570) appended to chain
main debug: looking for video filter module matching "gradient,chain": 46 candidates
main debug: using video filter module "gradient"
main debug: Filter 'gradient' (0000021fdd1ad680) appended to chain
main debug: using video filter module "chain"
main debug: Filter 'gradient' (0000021fdd1b06a0) appended to chain
main debug: Adding 'sepia' as interactive
main debug: looking for video filter module matching "sepia,chain": 46 candidates
main debug: using video filter module "sepia"
main debug: Filter 'sepia' (0000021fdd1b0a10) appended to chain
main debug: Adding a filter to compensate for format changes
main debug: looking for video converter module matching "any": 24 candidates
d3d11_filters debug: different 0 device p vs P
d3d11_filters warning: failed to read the 1 Display Adapter registry key (2)
d3d11_filters warning: can't find adapter in registry
main debug: looking for video converter module matching "any": 24 candidates
main debug: using video converter module "i420_nv12"
main debug: using video converter module "d3d11_filters"
main debug: Filter 'd3d11_filters' (0000021fdd1aee90) appended to chain
main debug: removing module "freetype"
main debug: looking for text renderer module matching "any": 3 candidates
main debug: VoutDisplayEvent 'resize' 2448x2048
direct3d11 debug: Detected size change 2448x2048
freetype debug: Using DWrite backend
freetype debug: DWrite_GetFamily(): family name: Arial
freetype debug: DWrite_ParseFamily(): using font at index 0 with weight 400 for bold: 0, italic: 0
freetype debug: DWrite_ParseFamily(): using font at index 1 with weight 700 for bold: 1, italic: 0
freetype debug: DWrite_ParseFamily(): using font at index 2 with weight 400 for bold: 0, italic: 1
freetype debug: DWrite_ParseFamily(): using font at index 3 with weight 700 for bold: 1, italic: 1
main debug: using text renderer module "freetype"
main debug: looking for hw decoder module matching "any": 2 candidates
d3d11va debug: different 0 device p vs P
d3d11va warning: failed to read the 1 Display Adapter registry key (2)
d3d11va warning: can't find adapter in registry
d3d11va debug: Reusing Direct3D11 device
d3d11va debug: CreateDevice succeed
d3d11va debug: - 'MPEG-2 variable-length decoder' is supported
d3d11va debug: - 'MPEG-2 inverse discrete cosine transform' is supported
d3d11va debug: - 'VC-1 variable-length decoder' is supported
d3d11va debug: - 'VC-1 variable-length decoder 2 (Intel)' is supported
d3d11va debug: - 'VC-1 variable-length decoder (Intel)' is supported
d3d11va debug: - 'Windows Media Video 9 IDCT' is supported
d3d11va debug: - 'VC-1 inverse discrete cosine transform' is supported
d3d11va debug: - 'H.264 variable-length decoder, no film grain technology' is supported
d3d11va debug: - 'H.264 stereo high profile, mbs flag set' is supported
d3d11va debug: - 'H.264 stereo high profile' is supported
d3d11va debug: - 'H.264 multiview high profile' is supported
d3d11va debug: - 'Unknown decoder 0xc528916c-0xc0af-0x4645-0x8cb2372b6d4adc2a' is supported
d3d11va debug: - 'Unknown decoder 0x91cd2d6e-0x897b-0x4fa1-0xb0d751dc88010e0a' is supported
d3d11va debug: - 'Unknown decoder 0x97688186-0x56a8-0x4094-0xb543fc9daaa49f4b' is supported
d3d11va debug: - 'Unknown decoder 0x1424d4dc-0x7cf5-0x4bb1-0x9cd7b63717a72a6b' is supported
d3d11va debug: - 'Unknown decoder 0xc346e8a3-0xcbed-0x4d27-0x87cca70eb4dc8c27' is supported
d3d11va debug: - 'Unknown decoder 0xffc79924-0x5eaf-0x4666-0xa73606190f281443' is supported
d3d11va debug: - 'Unknown decoder 0x464bdb3c-0x91c4-0x4e9b-0x896f225496ac4ed6' is supported
d3d11va debug: - 'Unknown decoder 0x9b31316b-0xf204-0x455d-0x8a8c9345dca77c01' is supported
d3d11va debug: - 'Unknown decoder 0xafe4285c-0xab63-0x4b2d-0x8278e6baacea2ce9' is supported
d3d11va debug: - 'Unknown decoder 0x277de9c5-0xed83-0x48dd-0xab8fac2d24b22943' is supported
d3d11va debug: - 'Unknown decoder 0x04c5ba10-0x4e9a-0x4b8e-0x8dbf4f4b48afa27c' is supported
d3d11va debug: - 'Unknown decoder 0x0acef8bc-0x285f-0x415d-0xab227bf2527a3d2e' is supported
d3d11va debug: - 'Unknown decoder 0x24d19fca-0xc5a2-0x4b8e-0x9f93f8f6ef15c890' is supported
d3d11va debug: - 'Unknown decoder 0x353aca91-0xd945-0x4c13-0xae7e469060fac8d8' is supported
d3d11va debug: - 'Unknown decoder 0x28566328-0xf041-0x4466-0x8b148f5831e78f8b' is supported
d3d11va debug: - 'Unknown decoder 0x6b4a94db-0x54fe-0x4ae1-0x9be47a7dad004600' is supported
d3d11va debug: - 'Unknown decoder 0xb8b28e0c-0xecab-0x4217-0x8c82eaaa9755aaf0' is supported
d3d11va debug: - 'Unknown decoder 0x8732ecfd-0x9747-0x4897-0xb42ae534f9ff2b7a' is supported
d3d11va debug: - 'Unknown decoder 0xe139b5ca-0x47b2-0x40e1-0xaf1cad71a67a1836' is supported
d3d11va debug: - 'Unknown decoder 0x056a6e36-0xf3a8-0x4d00-0x96637e9430358bf9' is supported
d3d11va debug: - 'Unknown decoder 0x5415a68c-0x231e-0x46f4-0x878b5e9a22e967e9' is supported
d3d11va debug: - 'Unknown decoder 0x161be912-0x44c2-0x49c0-0xb61ed946852b32a1' is supported
d3d11va debug: - 'Unknown decoder 0xd6d6bc4f-0xd51a-0x4712-0x97e8750917c860fd' is supported
d3d11va debug: - 'Unknown decoder 0x7fef652d-0x3233-0x44df-0xacf7ecfb584dab35' is supported
d3d11va debug: - 'Unknown decoder 0x580da148-0xe4bf-0x49b1-0x943b4214ab05a6ff' is supported
d3d11va debug: - 'Unknown decoder 0xcee393ab-0x1030-0x4f7b-0x8dbc55629c72f17e' is supported
d3d11va debug: - 'Unknown decoder 0x87b2ae39-0xc9a5-0x4c53-0x86b8a52d7edba488' is supported
d3d11va debug: - 'Unknown decoder 0x10e19ac8-0xbf39-0x4443-0xbec31b0cbfe4c7aa' is supported
d3d11va debug: - 'Unknown decoder 0x2dec00c7-0x21ee-0x4bf8-0x8f0e773f11f126a2' is supported
d3d11va debug: - 'Unknown decoder 0xc35153a0-0x23c0-0x4a81-0xb3bb6a1326f2b76b' is supported
d3d11va debug: - 'Unknown decoder 0xa33fd0ec-0xa9d3-0x4c21-0x9276c241cc90f6c7' is supported
d3d11va debug: - 'Unknown decoder 0x310e59d2-0x7ea4-0x47bb-0xb319500e78855336' is supported
d3d11va debug: - 'Unknown decoder 0x8c56eb1e-0x2b47-0x466f-0x8d337dbcd63f3df2' is supported
d3d11va debug: - 'HEVC Main profile' is supported
d3d11va debug: - 'Unknown decoder 0x75fc75f7-0xc589-0x4a07-0xa25b72e03b0383b3' is supported
d3d11va debug: - 'HEVC Main 10 profile' is supported
d3d11va debug: - 'Unknown decoder 0xe484dcb8-0xcac9-0x4859-0x99f55c0d45069089' is supported
d3d11va debug: - 'Unknown decoder 0x41a5af96-0xe415-0x4b0c-0x9d03907858e23e78' is supported
d3d11va debug: - 'Unknown decoder 0x6a6a81ba-0x912a-0x485d-0xb57fccd2d37b8d94' is supported
d3d11va debug: - 'Unknown decoder 0xe4e3cf5b-0x97d2-0x4658-0xaacb366e3ee2ceee' is supported
d3d11va debug: - 'Unknown decoder 0xfd9d9559-0x0fd3-0x4917-0xa9a707e714ee9ef9' is supported
d3d11va debug: - 'Unknown decoder 0x0e4bc693-0x5d2c-0x4936-0xb125aefe32b16d8a' is supported
d3d11va debug: - 'Unknown decoder 0x2f08b5b1-0xdbc2-0x4d48-0x883a4e7b8174cff6' is supported
d3d11va debug: - 'Unknown decoder 0x5467807a-0x295d-0x445d-0xbd2ecba8c2457c3d' is supported
d3d11va debug: - 'Unknown decoder 0xae0d4e15-0x2360-0x40a8-0xbf82028e6a0dd827' is supported
d3d11va debug: - 'Unknown decoder 0x8ff8a3aa-0xc456-0x4132-0xb6ef69d9dd72571d' is supported
d3d11va debug: - 'Unknown decoder 0xc23dd857-0x874b-0x423c-0xb6e082ceaa9b118a' is supported
d3d11va debug: - 'Unknown decoder 0x5b08e35d-0x0c66-0x4c51-0xa6f189d00cb2c197' is supported
d3d11va debug: - 'Unknown decoder 0x07cfaffb-0x5a2e-0x4b99-0xb62ae4ca53b6d5aa' is supported
d3d11va debug: - 'VP9 profile 0' is supported
d3d11va debug: - 'VP9 profile 2' is supported
d3d11va debug: - 'VP9 profile Intel' is supported
d3d11va debug: - 'Unknown decoder 0x68a21c7b-0xd58f-0x4e74-0x9993e4b8172b19a0' is supported
d3d11va debug: - 'Unknown decoder 0x80a3a7bd-0x89d8-0x4497-0xa2b82126af7e6eb8' is supported
d3d11va debug: - 'Unknown decoder 0x1d5c4d76-0xb55a-0x4430-0x904c3383a7ae3b16' is supported
d3d11va debug: - 'Unknown decoder 0xa7f759dd-0x5f54-0x4d7f-0x829142e883c546fe' is supported
d3d11va debug: - 'Unknown decoder 0xf34fa92f-0xdc79-0x474c-0xb0dbb7bd4522df77' is supported
d3d11va debug: - 'Unknown decoder 0xb8be4ccb-0xcf53-0x46ba-0x8d59d6b8a6da5d2a' is supported
d3d11va debug: - 'Unknown decoder 0xca44afc5-0xe1d0-0x42e6-0x9154b127186d4d40' is supported
d3d11va debug: - 'Unknown decoder 0xf9a16190-0x3fb4-0x4dc5-0x9846c8751f83d6d7' is supported
d3d11va debug: - 'Unknown decoder 0x50925b7b-0xe931-0x4978-0xa12a586630f095f9' is supported
d3d11va debug: - 'Unknown decoder 0xb69c20e0-0x2508-0x8790-0x0305875499e0a2d0' is supported
d3d11va debug: - 'Unknown decoder 0x49761bec-0x4b63-0x4349-0xa5ff87ffdf088466' is supported
d3d11va debug: Trying to use 'H.264 variable-length decoder, no film grain technology' as input
d3d11va debug: NV12 output is supported for decoder H.264 variable-length decoder, no film grain technology.
d3d11va debug: Using output format NV12 for decoder H.264 variable-length decoder, no film grain technology
d3d11va debug: va_pool_SetupDecoder id 27 2448x2048 count: 24
d3d11va debug: using external surface pool
d3d11va debug: ID3D11VideoDecoderOutputView succeed with 24 surfaces (2448x2048)
d3d11va debug: we got 9 decoder configurations
d3d11va debug: configuration[0] ConfigBitstreamRaw 2
d3d11va debug: configuration[1] ConfigBitstreamRaw 1
d3d11va debug: configuration[2] ConfigBitstreamRaw 2
d3d11va debug: configuration[3] ConfigBitstreamRaw 2
d3d11va debug: configuration[4] ConfigBitstreamRaw 2
d3d11va debug: configuration[5] ConfigBitstreamRaw 2
d3d11va debug: configuration[6] ConfigBitstreamRaw 2
d3d11va debug: configuration[7] ConfigBitstreamRaw 1
d3d11va debug: configuration[8] ConfigBitstreamRaw 1
d3d11va debug: DxCreateDecoderSurfaces succeed
main debug: using hw decoder module "d3d11va"
avcodec info: Using D3D11VA (Intel(R) UHD Graphics 750, vendor 8086(Intel), device 4c8a, revision 4) for hardware decoding
qt debug: Logical video size: 2448x2048
main debug: resized to 2448x2048
main debug: VoutDisplayEvent 'resize' 2448x2048
main debug: auto hiding mouse cursor
main warning: picture is too late to be displayed (missing 3891 ms)
main warning: picture is too late to be displayed (missing 6906 ms)
main warning: picture is too late to be displayed (missing 4938 ms)
main warning: picture is too late to be displayed (missing 4936 ms)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
main warning: picture is too late to be displayed (missing 8106 ms)
main warning: picture is too late to be displayed (missing 7228 ms)
main warning: picture is too late to be displayed (missing 8023 ms)
main warning: picture is too late to be displayed (missing 8041 ms)
main warning: picture is too late to be displayed (missing 7972 ms)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
main warning: picture is too late to be displayed (missing 11379 ms)
main warning: picture is too late to be displayed (missing 8455 ms)
main warning: picture is too late to be displayed (missing 11198 ms)
main warning: picture is too late to be displayed (missing 9201 ms)
main warning: picture is too late to be displayed (missing 10076 ms)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
main warning: picture is too late to be displayed (missing 11264 ms)
main warning: picture is too late to be displayed (missing 9342 ms)
main warning: picture is too late to be displayed (missing 11444 ms)
main warning: picture is too late to be displayed (missing 9518 ms)
main warning: picture is too late to be displayed (missing 9556 ms)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
main warning: picture is too late to be displayed (missing 9699 ms)
main warning: picture is too late to be displayed (missing 9582 ms)
main warning: picture is too late to be displayed (missing 9579 ms)
main warning: picture is too late to be displayed (missing 10576 ms)
main warning: picture is too late to be displayed (missing 14580 ms)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
main warning: picture is too late to be displayed (missing 8595 ms)
main warning: picture is too late to be displayed (missing 8544 ms)
main warning: picture is too late to be displayed (missing 15550 ms)
main warning: picture is too late to be displayed (missing 9535 ms)
main warning: picture is too late to be displayed (missing 9546 ms)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
main warning: picture is too late to be displayed (missing 9482 ms)
main warning: picture is too late to be displayed (missing 9532 ms)
main warning: picture is too late to be displayed (missing 9488 ms)
main warning: picture is too late to be displayed (missing 9533 ms)
main warning: picture is too late to be displayed (missing 9454 ms)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
main warning: picture is too late to be displayed (missing 8475 ms)
main warning: picture is too late to be displayed (missing 9420 ms)
main warning: picture is too late to be displayed (missing 7577 ms)
main warning: picture is too late to be displayed (missing 10465 ms)
main warning: picture is too late to be displayed (missing 7465 ms)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)
avcodec error: more than 5 seconds of late video -> dropping frame (computer too slow ?)

... I still don't know what exactly causes this problem, so I resize the source size smaller and finally works well.(2448x2048 to 200x100)

cv::VideoWriter writer;
writer.open(
  "appsrc ! "
  "video/x-raw,format=GRAY8,width=200,height=100,framerate=1/1 ! "
  "videoconvert ! "
  "openh264enc ! "
  "h264parse ! "
  "rtph264pay config-interval=10 pt=96 ! "
  "udpsink host=127.0.0.1 port=10000 ",
  cv::CAP_GSTREAMER,
  0, // fourcc
  1, // fps
  cv::Size(200, 100), 
  false // isColor
);
Solution

  • After trying some approaches, simply increasing FPS solved my problem.
    The bigger size of image was not the problem.
    Following code snippet is increased fps(to 20) version with minimum elements for streaming raw grayscale to RTP(even removing h264parse element has nothing to do with the result.).

    cv::VideoWriter writer;
writer.open(
  "appsrc ! "
  "video/x-raw,format=GRAY8,width=2448,height=2048,framerate=20/1 ! "
  "videoconvert ! "
  "openh264enc ! "
  "rtph264pay ! "
  "udpsink host=127.0.0.1 port=10000 ",
  cv::CAP_GSTREAMER,
  0, // fourcc
  20, // fps
  cv::Size(2448, 2048), // Inputting cv::Mat size 
  false // isColor
);

    Still didn't understand why the little fps(was 1 in my case) could be a problem.

    EDIT

    The following pipeline is my final server-side command. Adding h264parse and queue.

    "appsrc ! "
"video/x-raw,format=GRAY8,width=2448,height=2048,framerate=20/1 ! "
"videoconvert ! "
"queue ! "
"openh264enc ! "
"h264parse ! "
"rtph264pay ! "
"udpsink host=127.0.0.1 port=10000 "

    The following pipeline is client side-side command. There was almost no latency when running this pipeline for viewing the image stream.

    "udpsrc address=127.0.0.1 port=10000"
"caps=\"application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96\" !"
"rtph264depay !"
"h264parse !"
"openh264dec !"
"queue !"
"autovideosink"

# Command in shell: 
# gst-launch-1.0 udpsrc address=127.0.0.1 port=10000 caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! openh264dec ! queue ! autovideosink

    However, running .sdp file on VLC still 5-6 seconds of latency.