I'm currently creating a simple home surveillance system, where I would enter the app as an streamer or as a consumer.
Technologies
Only localhost is used for development purposes. I'm running both the client and server from a Windows' WSL terminal. The app is accessed from the Windows machine.
Current behaviour
After accessing to http://localhost/?stream I begin the streaming and wait for incoming messages from the websocket to start signaling with new peers.
When the consumer accesses http://localhost the WebRTC connection is performed and the tracks are received from the streamer side.
peerConnection.ontrack
event on the consumer side is actually triggered, but the media stream seems to only contain 1 video track which is muted.
On the streamer side when doing peerConnection.addTrack
I'm actually setting 2 tracks, one for the video and another for the audio.
Problem
When the consumer receives the tracks/stream from streamer, it sets it to the video.srcObject
property, but nothing happens. No video nor audio is played.
Expected behaviour
The video and audio from the streamer should be played on the consumer side after receiving the tracks/stream and setting it to the consumer's video.srcObject
property.
What I've tried
I have navigated the various related questions on StackOverflow, and I tried applying the various fixes which were suggested, but none of them worked properly.
video.srcObject
Code
I will post below the application files involved in the main application logic:
Don't worry, this repo is indeed a very basic and minimal example of the problem (it's not that massive).
Signaling Websocket service
import { Server } from "socket.io";
const io = new Server();
let streamer = null;
let users = new Set();
io.on("connect", (socket) => {
if (users.has(socket.id)) {
return;
}
console.log(
`Socket ${socket.id} connected - Client IP Address: ${socket.handshake.address}`
);
socket.on("disconnect", (reason) => {
console.log(`Socket: ${socket.id} disconnected - Reason: ${reason}`);
});
socket.on("begin-stream", () => {
console.log("begin stream", socket.id);
streamer = socket.id;
});
socket.on("request-start-stream", (data) => {
console.log("request-start-stream");
socket.to(streamer).emit("handle-request-start-stream", {
to: socket.id,
offer: data.offer,
});
});
socket.on("response-start-stream", (data) => {
console.log("response-start-stream");
socket.to(data.to).emit("handle-response-start-stream", data.answer);
});
});
io.listen(5432, { cors: true });
Streamer.tsx
import { useEffect } from "react";
import socket from "./socket";
import useVideo from "./useVideo";
export default function Streamer() {
const { videoRef, element: videoElement } = useVideo();
useEffect(() => {
init();
async function init() {
if (!videoRef.current) return;
const stream = await navigator.mediaDevices.getUserMedia({
video: true,
audio: true,
});
videoRef.current.srcObject = stream;
socket.emit("begin-stream");
socket.on("handle-request-start-stream", async ({ to, offer }) => {
const peerConnection = new RTCPeerConnection();
stream
.getTracks()
.forEach(
(track) =>
console.log("track", track) ||
peerConnection.addTrack(track, stream)
);
await peerConnection.setRemoteDescription(
new RTCSessionDescription(offer)
);
const answer = await peerConnection.createAnswer();
await peerConnection.setLocalDescription(
new RTCSessionDescription(answer)
);
socket.emit("response-start-stream", { to, answer });
});
}
}, []);
return videoElement;
}
Consumer.tsx
import { useEffect } from "react";
import socket from "./socket";
import useVideo from "./useVideo";
export default function Consumer() {
const { videoRef, element: videoElement } = useVideo();
useEffect(() => {
init();
async function init() {
const peerConnection = new RTCPeerConnection();
peerConnection.addEventListener("track", ({ streams: [stream] }) => {
if (!videoRef.current) return;
console.log('stream', stream)
videoRef.current.srcObject = stream;
});
const offer = await peerConnection.createOffer({
// workaround to receive the tracks
// if this is not specified, I will never receive the tracks
offerToReceiveAudio: true,
offerToReceiveVideo: true,
});
await peerConnection.setLocalDescription(
new RTCSessionDescription(offer)
);
socket.emit("request-start-stream", { offer });
socket.on("handle-response-start-stream", async (answer) => {
await peerConnection.setRemoteDescription(
new RTCSessionDescription(answer)
);
});
}
}, []);
return videoElement;
}
useVideo.tsx (video tag hook)
import { useRef } from "react";
export default function useVideo() {
const videoRef = useRef<HTMLVideoElement>(null);
const element = <video ref={videoRef} autoPlay />;
return { videoRef, element };
}
I have finally found the cause of the bug.
The ICE Candidate exchange logic was missing and I only had to add it.
Signaling service
socket.on("send-candidate", (data) => {
socket.to(data.to).emit("handle-send-candidate", data.candidate);
});
Streamer.tsx
peerConnection.addEventListener("icecandidate", (event) => {
if (event.candidate === null) {
return;
}
socket.emit("send-candidate", {
to,
candidate: event.candidate,
});
});
Consumer.tsx
socket.on("handle-send-candidate", (candidate) => {
peerConnection.addIceCandidate(candidate);
});