JavaCV FFmpegFrameGrabber preload audio

Question

I have an application that streams video data to a RTMP server using javacv's FFmpegFrameRecorder. I want to add some audio to this stream from a separate file - a short sound clip that I want to play on repeat.

Given the sound clip is very short, I want to preload the audio data into memory and just loop over it - so I can avoid excessive IO etc.

I've attempted to add audio to the stream using javacv's FFmpegFrameGrabber, as prescribed on multiple tutorials.

The addition of audio works perfectly if I don't attempt to preload/cache any of the audio data, for example:

private FFmpegFrameRecorder frameRecorder;
private FFmpegFrameGrabber frameGrabber;

...
//frameRecorder and frameGrabber setup during initialization
...

public void record(IplImage image) {
        try {
            frameRecorder.record(image);
            Frame frame = frameGrabber.grabFrame();
            if(frame == null) {
                frameGrabber = new FFmpegFrameGrabber("audioFileHere.wav");
                frameGrabber.start();
                frame = frameGrabber.grabFrame();
            }
            frameRecorder.record(frame);
        } catch (FrameRecorder.Exception e) {
            log.error(getMarker(FATAL), "Can't record frame!", e);
        } catch (FrameGrabber.Exception e) {
            log.error(getMarker(FATAL), "Can't record frame!", e);
        }
}

However, if I try to preload the audio data I get garbage sound being played:

private FFmpegFrameRecorder frameRecorder;
private List<FrameData> audioData;

private static final class FrameData {
    public final Buffer[] samples;
    public final Integer sampleRate;
    public final Integer audioChannels;

    //Constructors, getters and setters here
}

...
    //frameRecorder setup during initialization
    audioData = new ArrayList<>();
    FFmpegFrameGrabber audioGrabber = new FFmpegFrameGrabber("audioFileHere.wav");
    try {
        audioGrabber.start();
        Frame frame;
        while ((frame = audioGrabber.grabFrame()) != null) {
            Buffer[] buffers = frame.samples;
            Buffer[] copiedBuffers = new Buffer[buffers.length];
            for (int i = 0; i < buffers.length; i++) {
                copiedBuffers[i] = ((ShortBuffer) buffers[i]).duplicate();
            }
            FrameData frameData = new FrameData(copiedBuffers, frame.sampleRate, frame.audioChannels);
            audioData.add(frameData);
        }
    } catch (FrameGrabber.Exception e) {
        e.printStackTrace();
    }
...

private int frameCount = 0;

public void record(IplImage image) {
    frameCount++;
    try {
        FrameData frameData = audioData.get(frameCount % audioData.size());
        frameRecorder.record(image);
        frameRecorder.record(frameData.sampleRate, frameData.audioChannels, frameData.samples);
    } catch (FrameRecorder.Exception e) {
        log.error(getMarker(FATAL), "Can't record frame!", e);
    }
}

NOTE: I have to deep copy the Frame object because FFmpegFrameGrabber.grabFrame() recycles a single Frame object

Can someone explain why this doesn't work and/or how I could achieve the desired result?

Answer 1

I found the solution in my particular code example. ShortBuffer.duplicate() doesn't do a deep copy so it was causing issues. Instead I had to create a new ShortBuffer and copy over the contents.