Search code examples
javascriptiossafarimediarecordermediastream

Mobile Audio Tracks Not Working With MediaStream & MediaRecorder

I have a website (Angular 15 & Ionic 6) where a user uploads a video, then I apply some alterations to it on a canvas, capture the canvas stream, and output to a file

I first create a MediaStream of the video canvas that my alterations are occurring on along with the audio tracks from the video file captureStream or an AudioContext

Then I capture that MediaStream with a MediaRecorder and onstop write the data to a Blob

My current code works when uploading a file in Chrome from my Windows Desktop (many different file types) but when I choose a video on iOS, if I add audio tracks to the MediaStream then the MediaRecorder I capture it with has 0 bytes.

Can anyone help me figure out what I need to do to get the audio tracks added to my stream on Safari iOS?

This is my HTML (the source) and what I capture. Both have display: none because the user does not see the processing until later:

<video id="uploadedVideo" #uploadedVideo autoplay webkit-playsinline playsinline></video>
<canvas id="canvasVideo" #canvasVideo></canvas>

This is how I access these elements:

@ViewChild('uploadedVideo', { static: false }) videoElementRef: ElementRef<HTMLVideoElement>;
video: HTMLVideoElement;
@ViewChild('fileUpload', { static: false }) fileUpload: ElementRef;
@ViewChild('canvasVideo', { static: false }) videoCanvasElementRef: ElementRef<HTMLCanvasElement>;
videoCanvas: HTMLCanvasElement;
videoCanvasCtx: CanvasRenderingContext2D;

I put the gain of the video to 0 on init because I don't want the sound to play when you don't see the video:

silenceVideoAudio() {
    if (this.videoConnectedToAudioContext) {
      return;
    }
    this.audioContext = new AudioContext();
    this.sourceNode = this.audioContext.createMediaElementSource(this.video);
    const gainNode = this.audioContext.createGain();
    gainNode.gain.value = 0; // Ensure audio is silent
    this.sourceNode.connect(gainNode);
    gainNode.connect(this.audioContext.destination);
    this.videoConnectedToAudioContext = true;
}

When the file is selected I start the processing and set up for the MediaRecorder to stop when the video ends:

onVideoFileSelected(file: File) {
  this.video.src = URL.createObjectURL(file);
  this.silenceVideoAudio();
  this.video.onloadeddata = async () => {
    await this.startVideoProcessing();    };
  this.video.onended = async () => {
    this.recorder.stop();
  };
}

This is the startVideoProcessing where I get the audio tracks, set up the MediaStream and MediaRecorder and deal with the chunks when I'm done. There are no errors only 0 bytes on Safari:

async startVideoProcessing() {
    // Do some other setup that is unrelated

    let stream: MediaStream;
    if (this.isSafari) {
      stream = new MediaStream([...this.videoCanvas.captureStream(30).getVideoTracks()]);
    } else {
      stream = new MediaStream([...this.videoCanvas.captureStream(30).getVideoTracks(), this.getAudioTrack()]);
    }
    const options = {
        audioBitsPerSecond: 128000,
        videoBitsPerSecond: 2500000,
        mimeType: this.mimeType,
    };
    this.recorder = new MediaRecorder(stream, options);
    this.recorder.ondataavailable = (event) => {
        if (event.data && event.data.size > 0) {
            this.chunks.push(event.data);
        }
    };
    this.recorder.start();
    this.recorder.onerror = (event) => {
      console.error(event);
      throw new Error(event.toString());
    };
    this.recorder.onstop = () => {
      const videoBlob = new Blob(this.chunks, { type: this.mimeType });
      const videoFile = new File([videoBlob], this.mimeType === 'video/mp4' ? 'video-green-screen.mp4' : 'video-green-screen.webm', { type: this.mimeType });
      this.fileSelected.emit(videoFile);
    };
  }

As you can see in the conditional above where I check isSafari I do not add any audio track currently because if I do add it like in the else using this getAudioTrack method to get them like stream = new MediaStream([...this.videoCanvas.captureStream(30).getVideoTracks(), this.getAudioTrack()]); then the ondataavailable and this.chunks when the recorder stops is always empty. If I do not try to add the audio track then it is has the proper bytes. Adding the audio track like this works on every platform except Safari.

  getAudioTrack() {
    if ((this.video as any).captureStream) {
      this.videoAudioTracks = (this.video as any).captureStream().getAudioTracks()[0];
    } else if ((this.video as any).mozCaptureStream) {
      this.videoAudioTracks = (this.video as any).mozCaptureStream().getAudioTracks()[0];
    } else {
      const destination = this.audioContext.createMediaStreamDestination();
      this.sourceNode.connect(destination);
      this.videoAudioTracks = destination.stream.getAudioTracks()[0];
    }
    return this.videoAudioTracks;
  }

I am testing on Safari iOS 16.2

Can anyone help me alter these methods in a way that:

  1. Adds the audio track from the video to the canvas captured stream
  2. Works on iOS, Chrome, and Firefox (with support previous to iOS 16 if possible)

And help me understand why when I add an audio track to the MediaStream on Safari it makes all the MediaRecorder data empty

Solution

  • The code in the question works fine for setting up an AudioContext on Safari, the problem is that:

    1. A newly-created AudioContext will always begin in the suspended state, and a state change event will be fired whenever the state changes to a different state. This event is fired before the complete event is fired. (source: https://webaudio.github.io/web-audio-api/#dom-baseaudiocontext-onstatechange)
    2. Safari does not transition this state to running without user interaction (and in this case onVideoFileSelected is not direct enough). Safari also blocks playback in certain cases without user interaction. Read these two sections of this page for a breakdown on when and how Safari will deal with playing videos with and without audio and with and without user interaction: https://developer.apple.com/documentation/webkit/delivering_video_content_for_safari/#3030259) https://developer.apple.com/documentation/webkit/delivering_video_content_for_safari/#3030251

    Safari requires user interaction to change the state of a new AudioContext to running in this case. The call to resume or play the video must take place in the call stack of the button or element the user interacts with.

    In the case of this code break out the processing to happen as a result of a user clicking a button to process the video (at least on mobile or Safari):

      videoFile: File;

  onVideoFileSelected(file: File) {
    this.videoFile = file;
  }

  startVideoProcessing() {
    this.video.src = URL.createObjectURL(this.videoFile);
    this.silenceVideoAudio();
    this.video.onloadeddata = async () => {
      await this.videoProcessing();
      this.registerVideoFrameCallback();
    };
    this.video.onended = async () => {
      this.recorder.stop();
    };
  }

    <ion-button (click)="startVideoProcessing()">Process Video</ion-button>