Get MediaStream from Opentok.js subscriber

Question

I am using @opentok/client to build an Angular6 app for a simple multiuser video chat.

I want to do a 'spotlight' (eg. fullscreen the user currently speaking in the session) feature using hark.js for speech detection.

The problem is that this library needs the source MediaStream to process the audio and emit events that I would catch and process.

I would like to get the MediaStream from OpenTok's subscriber class, but how to do so is nowhere to be found, but since it's based on common WebRTC it should be doable.

Any leads?

Sample code :

 this.opentokService.initSession().then((session: OT.Session) => {
  this.session = session;
  this.session.on('streamCreated', (event) => {
    this.streams.push(event.stream); //creates the subscribers elements
    var speechEvents = hark(event.stream, {}); //raises error, how to get a proper MediaStream here?
  });

Answer 1

You are right, because OpenTok is just common WebRTC you can get the MediaStream from the underlying video element in the subscriber like so:

const mediaStream = subscriber.element.querySelector('video').srcObject;

This will work in recent versions of Chrome, Firefox, Safari and Edge. It will not work in Internet Explorer because it uses a plugin and does not have MediaStream objects anyway.

I also want to point out that the Subscriber itself has audioLevelUpdated events that could be used to achieve the same thing. The event gives you an audioLevel between 0 and 1. There is a good example in the documentation for how to convert this to a dB range https://tokbox.com/developer/sdks/js/reference/Subscriber.html#events

I'm not sure whether hark.js does some processing to make sure it is speech, if so it might still work better. The audio level event is just based on noise regardless of whether or not it is speech.