Search code examples
javascriptwebrtcvideo-streaming

WebRTC fails to play the stream from remote peer after setting the video.srcObject property


I'm currently creating a simple home surveillance system, where I would enter the app as an streamer or as a consumer.

Technologies

  • socket.io - for signaling between streamer and connected peer
  • basic vite app - react
  • webrtc to create a streaming channel between 2 peers

Only localhost is used for development purposes. I'm running both the client and server from a Windows' WSL terminal. The app is accessed from the Windows machine.

Current behaviour

After accessing to http://localhost/?stream I begin the streaming and wait for incoming messages from the websocket to start signaling with new peers.

When the consumer accesses http://localhost the WebRTC connection is performed and the tracks are received from the streamer side.

peerConnection.ontrack event on the consumer side is actually triggered, but the media stream seems to only contain 1 video track which is muted.

On the streamer side when doing peerConnection.addTrack I'm actually setting 2 tracks, one for the video and another for the audio.

Problem

When the consumer receives the tracks/stream from streamer, it sets it to the video.srcObject property, but nothing happens. No video nor audio is played.

Expected behaviour

The video and audio from the streamer should be played on the consumer side after receiving the tracks/stream and setting it to the consumer's video.srcObject property.

What I've tried

I have navigated the various related questions on StackOverflow, and I tried applying the various fixes which were suggested, but none of them worked properly.

  • setting the video element autoPlay, playsInline and muted to true
  • playing the video element programatically after setting the remote video.srcObject
  • using a STUN server - but I believe that's not a requirement because I'm using localhost and not connecting 2 peers on the wide area network / internet.
  • connecting the peers asynchronously after clicking a button on the consumer side - to not to try to play a video without user interaction.

Code

I will post below the application files involved in the main application logic:

Don't worry, this repo is indeed a very basic and minimal example of the problem (it's not that massive).

Signaling Websocket service

import { Server } from "socket.io";

const io = new Server();

let streamer = null;
let users = new Set();

io.on("connect", (socket) => {
  if (users.has(socket.id)) {
    return;
  }

  console.log(
    `Socket ${socket.id} connected - Client IP Address: ${socket.handshake.address}`
  );

  socket.on("disconnect", (reason) => {
    console.log(`Socket: ${socket.id} disconnected - Reason: ${reason}`);
  });

  socket.on("begin-stream", () => {
    console.log("begin stream", socket.id);
    streamer = socket.id;
  });

  socket.on("request-start-stream", (data) => {
    console.log("request-start-stream");
    socket.to(streamer).emit("handle-request-start-stream", {
      to: socket.id,
      offer: data.offer,
    });
  });

  socket.on("response-start-stream", (data) => {
    console.log("response-start-stream");
    socket.to(data.to).emit("handle-response-start-stream", data.answer);
  });
});

io.listen(5432, { cors: true });

Streamer.tsx

import { useEffect } from "react";

import socket from "./socket";
import useVideo from "./useVideo";

export default function Streamer() {
  const { videoRef, element: videoElement } = useVideo();

  useEffect(() => {
    init();

    async function init() {
      if (!videoRef.current) return;

      const stream = await navigator.mediaDevices.getUserMedia({
        video: true,
        audio: true,
      });

      videoRef.current.srcObject = stream;

      socket.emit("begin-stream");

      socket.on("handle-request-start-stream", async ({ to, offer }) => {
        const peerConnection = new RTCPeerConnection();
        stream
          .getTracks()
          .forEach(
            (track) =>
              console.log("track", track) ||
              peerConnection.addTrack(track, stream)
          );

        await peerConnection.setRemoteDescription(
          new RTCSessionDescription(offer)
        );

        const answer = await peerConnection.createAnswer();
        await peerConnection.setLocalDescription(
          new RTCSessionDescription(answer)
        );
        socket.emit("response-start-stream", { to, answer });
      });
    }
  }, []);

  return videoElement;
}

Consumer.tsx

import { useEffect } from "react";

import socket from "./socket";
import useVideo from "./useVideo";

export default function Consumer() {
  const { videoRef, element: videoElement } = useVideo();

  useEffect(() => {
    init();

    async function init() {
      const peerConnection = new RTCPeerConnection();
      peerConnection.addEventListener("track", ({ streams: [stream] }) => {
        if (!videoRef.current) return;
        console.log('stream', stream)

        videoRef.current.srcObject = stream;
      });

      const offer = await peerConnection.createOffer({
        // workaround to receive the tracks
        // if this is not specified, I will never receive the tracks
        offerToReceiveAudio: true,
        offerToReceiveVideo: true,
      });
      await peerConnection.setLocalDescription(
        new RTCSessionDescription(offer)
      );
      socket.emit("request-start-stream", { offer });

      socket.on("handle-response-start-stream", async (answer) => {
        await peerConnection.setRemoteDescription(
          new RTCSessionDescription(answer)
        );
      });
    }
  }, []);

  return videoElement;
}

useVideo.tsx (video tag hook)

import { useRef } from "react";

export default function useVideo() {
  const videoRef = useRef<HTMLVideoElement>(null);

  const element = <video ref={videoRef} autoPlay />;

  return { videoRef, element };
}

Solution

  • I have finally found the cause of the bug.

    The ICE Candidate exchange logic was missing and I only had to add it.

    Signaling service

    socket.on("send-candidate", (data) => {
      socket.to(data.to).emit("handle-send-candidate", data.candidate);
    });
    

    Streamer.tsx

    peerConnection.addEventListener("icecandidate", (event) => {
      if (event.candidate === null) {
        return;
      }
    
      socket.emit("send-candidate", {
        to,
        candidate: event.candidate,
      });
    });
    

    Consumer.tsx

    socket.on("handle-send-candidate", (candidate) => {
      peerConnection.addIceCandidate(candidate);
    });