Webcam api : Maximizing the 720p capture

Question

I am attempting to capture a video recording through an external camera, Logitec C922. Using java, I can make this possible through webcam api.

    <dependency>
        <groupId>com.github.sarxos</groupId>
        <artifactId>webcam-capture</artifactId>
        <version>0.3.10</version>
    </dependency>
    <dependency>
        <groupId>xuggle</groupId>
        <artifactId>xuggle-xuggler</artifactId>
        <version>5.4</version>
    </dependency>

However, for the life of me, I cannot make it record at 60FPS. The video randomly stutters when stored, and is not smooth at all.

I can connect to the camera, using the following details.

final List<Webcam> webcams = Webcam.getWebcams();

for (final Webcam cam : webcams) {
    if (cam.getName().contains("C922")) {
        System.out.println("### Logitec C922 cam found");
        webcam = cam;
        break;
    }
}

I set the size of the cam to the following:

final Dimension[] nonStandardResolutions = new Dimension[] { WebcamResolution.HD720.getSize(), };
webcam.setCustomViewSizes(nonStandardResolutions);
webcam.setViewSize(WebcamResolution.HD720.getSize());
webcam.open(true);

And then I capture the images:

while (continueRecording) {
    // capture the webcam image
    final BufferedImage webcamImage = ConverterFactory.convertToType(webcam.getImage(),
                    BufferedImage.TYPE_3BYTE_BGR);
    final Date timeOfCapture = new Date();

    // convert the image and store
    final IConverter converter = ConverterFactory.createConverter(webcamImage, IPixelFormat.Type.YUV420P);
    final IVideoPicture frame = converter.toPicture(webcamImage,
                    (System.currentTimeMillis() - start) * 1000);

    frame.setKeyFrame(false);
    frame.setQuality(0);
    writer.encodeVideo(0, frame);

}

My writer is defined as follows:

final Dimension size = WebcamResolution.HD720.getSize();
final IMediaWriter writer = ToolFactory.makeWriter(videoFile.getName());
writer.addVideoStream(0, 0, ICodec.ID.CODEC_ID_H264, size.width, size.height);

I am honestly not sure what in my code could be causing this. Given that I lower the resolution, I get no problems. ( 480p ) Could the issue be with the codes I am using?

Answer 1

As some of the comments mentioned, introducing queues does solve the problem. Here is the general logic to performs the needed steps. Note, I've setup my code for a lower resolution, as it allows me to capture 100FPS per sec. Adjust as needed.

Class to link the image/video capture and class to edit it :

public class WebcamRecorder {

    final Dimension size = WebcamResolution.QVGA.getSize();
    final Stopper stopper = new Stopper();

    public void startRecording() throws Exception {

        final Webcam webcam = Webcam.getDefault();
        webcam.setViewSize(size);
        webcam.open(true);

        final BlockingQueue<CapturedFrame> queue = new LinkedBlockingQueue<CapturedFrame>();
        final Thread recordingThread = new Thread(new RecordingThread(queue, webcam, stopper));
        final Thread imageProcessingThread = new Thread(new ImageProcessingThread(queue, size));

        recordingThread.start();
        imageProcessingThread.start();
    }

    public void stopRecording() {
        stopper.setStop(true);
    }

}

RecordingThread :

public void run() {
    try {
        System.out.println("## capturing images began");
        while (true) {
            final BufferedImage webcamImage = ConverterFactory.convertToType(webcam.getImage(),
                    BufferedImage.TYPE_3BYTE_BGR);
            final Date timeOfCapture = new Date();
            queue.put(new CapturedFrame(webcamImage, timeOfCapture, false));
            if (stopper.isStop()) {
                System.out.println("### signal to stop capturing images received");
                queue.put(new CapturedFrame(null, null, true));
                break;
            }
        }
    } catch (InterruptedException e) {
        System.out.println("### threading issues during recording:: " + e.getMessage());
    } finally {
        System.out.println("## capturing images end");
        if (webcam.isOpen()) {
            webcam.close();
        }
    }
}

ImageProcessingThread:

public void run() {
    writer.addVideoStream(0, 0, ICodec.ID.CODEC_ID_H264, size.width, size.height);
    try {
        int frameIdx = 0;
        final long start = System.currentTimeMillis();
        while (true) {
            final CapturedFrame capturedFrame = queue.take();
            if (capturedFrame.isEnd()) {
                break;
            }
            final BufferedImage webcamImage = capturedFrame.getImage();
            size.height);

            // convert the image and store
            final IConverter converter = ConverterFactory.createConverter(webcamImage, IPixelFormat.Type.YUV420P);
            final long end = System.currentTimeMillis();
            final IVideoPicture frame = converter.toPicture(webcamImage, (end - start) * 1000);

            frame.setKeyFrame((frameIdx++ == 0));
            frame.setQuality(0);
            writer.encodeVideo(0, frame);
        }
    } catch (final InterruptedException e) {
        System.out.println("### threading issues during image processing:: " + e.getMessage());
    } finally {
        if (writer != null) {
            writer.close();
        }
    }

The way it works is pretty simple. The WebcamRecord class creates an instance of a queue that is shared between the video capture and the image processing. The RecordingThread sends bufferedImages to the queue ( in my case, it a pojo, called CapturedFrame ( which has a BufferedImage in it ) ). The ImageProcessingThread will listen and pull data from the queue. If it does not receive a signal that the writing should end, the loop never dies.