I'm working on a project that takes individual images from an RTSP-Stream and manipulates them (drawing bounding boxes). Those images should be restreamed (h264 encoded) on a separate RTSP-stream on an other address and shouldn't be saved on the local disk.
My current code so far is:
{
// OpenCV VideoCapture: Sample RTSP-Stream
var capture = new VideoCapture("rtsp://195.200.199.8/mpeg4/media.amp");
capture.Set(VideoCaptureProperties.FourCC, FourCC.FromFourChars('M', 'P', 'G', '4'));
var mat = new Mat();
// LibVlcSharpStreamer
Core.Initialize();
var libvlc = new LibVLC();
var player = new MediaPlayer(libvlc);
player.Play();
while (true)
{
if (capture.Grab())
{
mat = capture.RetrieveMat();
// Do some manipulation in here
var media = new Media(libvlc, new StreamMediaInput(mat.ToMemoryStream(".jpg")));
media.AddOption(":no-audio");
media.AddOption(":sout=#transcode{vcodec=h264,fps=10,vb=1024,acodec=none}:rtp{mux=ts,sdp=rtsp://192.168.xxx.xxx:554/video}");
media.AddOption(":sout-keep");
player.Media = media;
// Display screen
Cv2.ImShow("image", mat);
Cv2.WaitKey(1);
}
}
}
It is a little bit messy, because of testing purposes, but it works if I just use the given RTSP-Stream as the Media instead of the fetched images. I have some success with piping the images (as bytes) into the cvlc command line (python get_images.py | cvlc -v --demux=rawvideo --rawvid-fps=25 --rawvid-chroma=RV24 --sout '#transcode{vcodec=h264,fps=25,vb=1024,acodec=none}:rtp{sdp="rtsp://:554/video"}'), but it should be integrated in c#. get_images.py just reads images in a while-loop, wirtes a text on it and forwards them into std-out.
My thoughts on solving this problem is, to input the images via the StreamMediaInput-class and to dynamically change the media, if a new image has been retrieved. But it doesn't work, nothing can be seen with VLC or FFPlay.
Does someone has faced a similar Problem? How can the StreamMediaInput-Object can be changed dynamically, such that new images are broadcastet correctly?
Thank you for taking the time to read this post. Have a nice day!
EDIT:
I tried to implement my own MediaInput class (very similar to MemoryStramMediaInput) with the modification of UpdateMemoryStream(). The MemoryStream gets updated by every new retrieved image, but the read() will not get called a second time (read() is called once per Medium). I am trying to implement the blocking read(), but I am struggeling to find a good way in implementing it. The code so far is:
EDIT 2:
I decided to implement the blocking with an ManualResetEvent, which blocks the read(), if the Position is at the end of the Stream. Futhermore the read is looped in a while to keep the data in the stream updated. It still does not work. My Code so far:
using System;
using System.Collections.Generic;
using System.IO;
using System.Text;
using System.Threading;
using LibVLCSharp.Shared;
namespace LibVlcSharpStreamer
{
/// <summary>
/// A <see cref="MediaInput"/> implementation that reads from a .NET stream
/// </summary>
public class MemoryStreamMediaInput : MediaInput
{
private Stream _stream;
private ManualResetEvent manualResetEvent = new ManualResetEvent(false);
#if NET40
private readonly byte[] _readBuffer = new byte[0x4000];
#endif
/// <summary>
/// Initializes a new instance of <see cref="StreamMediaInput"/>, which reads from the given .NET stream.
/// </summary>
/// <remarks>You are still responsible to dispose the stream you give as input.</remarks>
/// <param name="stream">The stream to be read from.</param>
public MemoryStreamMediaInput(Stream stream)
{
_stream = stream ?? throw new ArgumentNullException(nameof(stream));
CanSeek = stream.CanSeek;
}
/// <summary>
/// Initializes a new instance of <see cref="StreamMediaInput"/>, which reads from the given .NET stream.
/// </summary>
/// <remarks>You are still responsible to dispose the stream you give as input.</remarks>
/// <param name="stream">The stream to be read from.</param>
public void UpdateMemoryStream(Stream stream)
{
stream.CopyTo(_stream);
_stream.Position = 0;
manualResetEvent.Set();
manualResetEvent.Reset();
Console.WriteLine("released");
}
/// <summary>
/// LibVLC calls this method when it wants to open the media
/// </summary>
/// <param name="size">This value must be filled with the length of the media (or ulong.MaxValue if unknown)</param>
/// <returns><c>true</c> if the stream opened successfully</returns>
public override bool Open(out ulong size)
{
try
{
try
{
size = (ulong)_stream.Length;
}
catch (Exception)
{
// byte length of the bitstream or UINT64_MAX if unknown
size = ulong.MaxValue;
}
if (_stream.CanSeek)
{
_stream.Seek(0L, SeekOrigin.Begin);
}
return true;
}
catch (Exception)
{
size = 0UL;
return false;
}
}
/// <summary>
/// LibVLC calls this method when it wants to read the media
/// </summary>
/// <param name="buf">The buffer where read data must be written</param>
/// <param name="len">The buffer length</param>
/// <returns>strictly positive number of bytes read, 0 on end-of-stream, or -1 on non-recoverable error</returns>
public unsafe override int Read(IntPtr buf, uint len)
{
try
{
while (_stream.CanSeek)
{
if (_stream.Position >= _stream.Length)
{
manualResetEvent.WaitOne();
}
var read = _stream.Read(new Span<byte>(buf.ToPointer(), (int)Math.Min(len, int.MaxValue)));
// Debug Purpose
Console.WriteLine(read);
}
return -1;
}
catch (Exception)
{
return -1;
}
}
/// <summary>
/// LibVLC calls this method when it wants to seek to a specific position in the media
/// </summary>
/// <param name="offset">The offset, in bytes, since the beginning of the stream</param>
/// <returns><c>true</c> if the seek succeeded, false otherwise</returns>
public override bool Seek(ulong offset)
{
try
{
_stream.Seek((long)offset, SeekOrigin.Begin);
return true;
}
catch (Exception)
{
return false;
}
}
/// <summary>
/// LibVLC calls this method when it wants to close the media.
/// </summary>
public override void Close()
{
try
{
if (_stream.CanSeek)
_stream.Seek(0, SeekOrigin.Begin);
}
catch (Exception)
{
// ignored
}
}
}
}
I have marked the spot in the code, where the blocking clause would fit well, in my opinion.
After a long time trying to implement a good solution with no success, we used the method with piping the retrieved Mat into Vlc.exe. To anybody having the same issue:
Use ProcessStartInfo to specify the path of vlc. The images can be piped into the process with Process.StandardInput.BaseStream.Write(). Use a loop to capture the images periodically (e.g. with OpenCV).