Streamlink Streamdata processing in OpenCV Python

Question

I'm trying to read a twitch stream via streamlink (https://streamlink.github.io/api_guide.html) into OpenCV for further processing in Python.

What works, reading the stream into a stream.ts file via popen and then into opencv:

import subprocess
import os
import time

def create_new_streaming_file(stream_filename="stream0", stream_link="https://www.twitch.tv/tsm_viss"):

            try:
                os.remove('./Engine/streaming_util/'+stream_filename+'.ts')
            except OSError:
                pass

            cmd = "streamlink --force --output ./Engine/streaming_util/"+stream_filename+".ts "+stream_link+" best"
            subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

create_new_streaming_file()

video_capture = cv2.VideoCapture('./Engine/streaming_util/stream0.ts')

Which is very slow and the stream stops after about 30 seconds.

I would like to read the bytestream directly into openCV out of streamlink's python api. What works is printing out the latest n bytes of a stream into the console:

import streamlink

streams = streamlink.streams("https://www.twitch.tv/grimmmz")
stream = streams["best"]
fd = stream.open()

while True:
    data = fd.read(1024)
    print(data)

I'm looking for something like this (does not work but you'll get the concept):

streams = streamlink.streams("https://www.twitch.tv/grimmmz")
stream = streams["best"]
fd = stream.open()
bytes=''


while True:
# to read mjpeg frame -
    bytes+= fd.read(1024)

    a = bytes.find('\xff\xd8')
    b = bytes.find('\xff\xd9')

    if a!=-1 and b!=-1:
        jpg = bytes[a:b+2]
        bytes= bytes[b+2:]
        img = cv2.imdecode(np.fromstring(jpg, dtype=np.uint8),cv2.CV_LOAD_IMAGE_COLOR)
        cv2.imwrite('messigray.png', img)
        cv2.imshow('cam2', img)

    else:
        continue

Thanks a lot in advance!

Answer 1

It was quite tricky to accomplish with reasonable performance, though.

Check out the project: https://github.com/DanielTea/rage-analytics/blob/master/README.md

The main file is realtime_Videostreamer.py in the engine folder. If you initialize this object it creates an ffmpeg subprocess and fills a queue with video frames in an extra thread. This architecture prevents the mainthread from blocking so depending on your networkspeed and cpu power a couple streams can be analyzed in parallel.

This solution works with twitch streams very well. Didn’t try out other streaming sites.

More info about this project.