Rendering video to file that may have an inconsistent frame rate

Question

I'm getting raw video frames from a source (that can be considered a black box) at a rate that can be inconsistent. I'm trying to record the video feed to the disk. I'm doing so with AForge's VideoRecorder and am writing to an MP4 file.

However, the inconsistent rate at which I receive frames causes the video to appear sped up. It seems that I only have the ability to create video files that have a fixed frame rate, even though the source does not have a fixed frame rate.

This isn't an issue when rendering to the screen, as we can just render as fast as possible. I can't do this when writing to the file, since playing back the file would play at the fixed frame rate.

What solutions are there? The output does not have to be the same video format as long as there's some reasonable way to convert it later (which wouldn't have to be real time). The video feeds can be quite long, so I can't just store everything in memory and encode later.

My code currently looks along the lines of:

VideoFileWriter writer = new VideoFileWriter();
Stopwatch stopwatch = new Stopwatch();

public override void Start() {
    writer.Open("output.mp4", videoWidth, videoHeight, frameRate, AForge.Video.FFMPEG.VideoCodec.MPEG4);
    stopwatch.Start();
}

public override void End() {
    writer.Close();
}

public override void Draw(Frame frame) {
    double elapsedTimeInSeconds = stopwatch.ElapsedTicks / (double) Stopwatch.Frequency;
    double timeBetweenFramesInSeconds = 1.0 / FrameRate;
    if (elapsedTimeInSeconds >= timeBetweenFramesInSeconds) {
        stopwatch.Restart();
        writer.WriteVideoFrame(frame.ToBitmap());
    }
}

Where our black box calls the Start, End, and Draw methods. The current check that I have in Draw prevents us from drawing too fast, but doesn't do anything to handle the case of drawing too slowly.

Answer 1

It turns out WriteVideoFrame is overloaded and one variant of the function is WriteVideoFrame(Bitmap frame, TimeSpan timestamp). As you can guess, the time stamp is used to make a frame appear at a certain time in the video.

Thus, by keeping track of the real time, we can set each frame to use the time that it should be in the video. Of course, the video quality will be worse if you can't render quickly enough, but this resolves the issue at hand.

Here's the code that I used for the Draw function:

// We can provide a frame offset so that each frame has a time that it's supposed to be
// seen at. This ensures that the video looks correct if the render rate is lower than
// the frame rate, since these times will be used (it'll be choppy, but at least it'll
// be paced correctly -- necessary so that sound won't go out of sync).
long currentTick = DateTime.Now.Ticks;
StartTick = StartTick ?? currentTick;
var frameOffset = new TimeSpan(currentTick - StartTick.Value);

// Figure out if we need to render this frame to the file (ie, has enough time passed
// that this frame will be in the file?). This prevents us from going over the
// desired frame rate and improves performance (assuming that we can go over the frame
// rate).
double elapsedTimeInSeconds = stopwatch.ElapsedTicks / (double) Stopwatch.Frequency;
double timeBetweenFramesInSeconds = 1.0 / FrameRate;
if (elapsedTimeInSeconds >= timeBetweenFramesInSeconds)
{
    stopwatch.Restart();

    Writer.WriteVideoFrame(frame.ToBitmap(), frameOffset);
}

Where StartTick is a long? member of the object.