can't pipe in numpy arrays (images) to ffmpeg subprocess in python

Question

I'm trying to capture webcam video stream using opencv and pipe raw frames into ffmpeg subprocess, apply 3d .cube lut, bring back those lut applied frames into opencv and display it using cv2.imshow.

This is my code:

import cv2
import subprocess as sp
import numpy as np

lut_cmd = [
            'ffmpeg', '-f', 'rawvideo', '-pixel_format', 'bgr24', '-s', '1280x720', '-framerate', '30', '-i', '-', '-an', '-vf',
            'lut3d=file=lut/luts/lut.cube', '-f', 'rawvideo', 'pipe:1'
        ]

lut_process = sp.Popen(lut_cmd, stdin=sp.PIPE, stdout=sp.PIPE)

width = 1280
height = 720

video_capture = cv2.VideoCapture(0)

while True:
    ret, frame = video_capture.read()

    if not ret:
        break
 
    # Write raw video frame to input stream of ffmpeg sub-process.
    lut_process.stdin.write(frame.tobytes())
    lut_process.stdin.flush()
    print("flushed")

    # Read the processed frame from the ffmpeg subprocess
    raw_frame = lut_process.stdout.read(width * height * 3)
    print("read")
    frame = np.frombuffer(raw_frame, dtype=np.uint8).reshape(height, width, 3)

    cv2.imshow('Video', frame)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

lut_process.terminate()
video_capture.release()

cv2.destroyAllWindows()

code gets stuck at reading from ffmpeg part: raw_frame = lut_process.stdout.read(width * height * 3)

this is what i get when i run the code:

flushed
Input #0, rawvideo, from 'fd:':
  Duration: N/A, start: 0.000000, bitrate: 663552 kb/s
  Stream #0:0: Video: rawvideo (BGR[24] / 0x18524742), bgr24, 1280x720, 663552 kb/s, 30 tbr, 30 tbn
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> rawvideo (native))
Output #0, rawvideo, to 'pipe:1':
  Metadata:
    encoder         : Lavf60.3.100
  Stream #0:0: Video: rawvideo (BGR[24] / 0x18524742), bgr24(progressive), 1280x720, q=2-31, 663552 kb/s, 30 fps, 30 tbn
    Metadata:
      encoder         : Lavc60.3.100 rawvideo
frame=    0 fps=0.0 q=0.0 size=       0kB time=-577014:32:22.77 bitrate=  -0.0kbits/s speed=N/A  

"read" never gets printed. ffmpeg is stuck at 0fps. cv2.imshow doesn't show up.

I tried changing lut_process.stdin.write(frame.tobytes()) to lut_process.stdin.write(frame.tostring()), but result was same.

I tried adding 3 seconds pause before first write to ffmpeg begin, thinking maybe ffmpeg was not ready to process frames, but result was same.

I'm sure that my webcam is working, and I know it's video stream is 1280x720 30fps.

I was successful at Displaying webcam stream just using opencv, set FFmpeg input directly to my webcam and get output result using stdout.read, displaying it using opencv.

have no idea what should I try next.

I am using macOS 12.6, openCV 4.7.0, ffmpeg 6.0, python 3.10.11, and visual studio code.

Any help would be greatly appreciated.

Answer 1

This is not my cleanest, or tidiest piece of code, but I have got something working and wanted to share it with you - I may clean it up later. I think the issue is that ffmpeg and Python subprocesses don't play that well together and there is some buffering going on and chances of deadlock. So, I abstracted out the reading of video frames from the camera and feeding them into ffmpeg as a separate thread and then it all works. It needs tidying up and improvements to error handling and the user pressing q to quit.

#!/usr/bin/env python3

import cv2
import subprocess as sp
import numpy as np
import sys
import time
import os
import threading

width = 1280
height = 720

def pumpFFMPEG(fd):
    """Read frames from camera and pump into ffmpeg."""
    video_capture = cv2.VideoCapture(0)

    while True:
        ret, frame = video_capture.read()
        frame = cv2.resize(frame, (width,height))
        fd.write(frame.tobytes())
    video_capture.release()
        
lut_cmd = [
            'ffmpeg', '-nostdin', '-loglevel', 'error', '-f', 'rawvideo', '-pixel_format', 'bgr24', '-video_size', '1280x720', '-i', '-', '-framerate', '30', '-an', '-vf',
            'lut3d=file=invert.cube', '-f', 'rawvideo', 'pipe:1'
        ]
lut_process = sp.Popen(lut_cmd, bufsize=width*height*3,stdin=sp.PIPE, stdout=sp.PIPE)


thr = threading.Thread(target=pumpFFMPEG, args=(lut_process.stdin,))
thr.start()

while True:
 
    # Read the processed frame from the ffmpeg subprocess
    raw_frame = lut_process.stdout.read(width*height*3)
    frame = np.frombuffer(raw_frame, dtype=np.uint8).reshape(height, width, 3)
    cv2.imshow('Video', frame)
    cv2.waitKey(1)

cv2.destroyAllWindows()

Some of the parameters to ffmpeg are maybe unnecessary, so you can try removing them one-by-one till it stops working. You may also want to use stderr=sp.DEVNULL.

---

I also made a 3dLUT. Normally, you would make a HALD CLUT with ImageMagick like this:

magick hald:8 input.png

Then you would apply your Lightroom processing to input.png and save it as output.png. Then you need to generate a 3dCLUT that implements that processing - I did that with this.

The command to generate a cube LUT for ffmpeg would be:

./HALDtoCUBE3DLUT.py output.png LUT.cube

---

Rather than go through all that palaver with Lightroom/Photoshop, I just made a LUT and inverted it all in one go with ImageMagick:

magick hald:8 -negate output.png
./HALDtoCUBE3DLUT.py output.png invert.cube

Note that the inversion I am referring to is brightness inversion, i.e. "black becoming white" rather than physical inversion, i.e. "top becoming bottom".