Procedural Audio using MediaStreamTrack

Question

I want to encode a video (from a canvas) and add procedural audio to it.

The encoding can be accomplished with MediaRecorder that takes a MediaStream.

For the stream, I want to obtain the video part from a canvas, using the canvas.captureStream() call.

I want to add an audio track to the stream. But instead of microphone input, I want to generate the samples for those on the fly, for simplicity sake, let's assume it writes out a sine-wave.

How can I create a MediaStreamTrack that generates procedural audio?

Answer 1

The Web Audio API has a createMediaStreamDestination() method, which will return a MediaStreamAudioDestinationNode object, on which you'll be able to connect your audio context, and which will give you access to a MediaStream instance fed by the audio context audio output.

document.querySelector("button").onclick = (evt) => {
  const duration = 5;
  evt.target.remove();
  const audioContext = new AudioContext();
  const osc = audioContext.createOscillator();

  
  const destNode = audioContext.createMediaStreamDestination();
  const { stream } = destNode;
  
  osc.connect(destNode);
  osc.connect(audioContext.destination);
  osc.start(0);  
  osc.frequency.value = 80;
  osc.frequency.exponentialRampToValueAtTime(440, audioContext.currentTime+10);
  osc.stop(duration);
  
  // stream.addTrack(canvasStream.getVideoTracks()[0]);
  const recorder = new MediaRecorder(stream);
  const chunks = [];
  recorder.ondataavailable = ({data}) => chunks.push(data);
  recorder.onstop = (evt) => {
    const el = new Audio();
    const [{ type }] = chunks; // for Safari
    el.src = URL.createObjectURL(new Blob(chunks, { type }));
    el.controls = true;
    document.body.append(el);
  };
  recorder.start();
  setTimeout(() => recorder.stop(), duration * 1000);
  console.log(`Started recording, please wait ${duration}s`);
 };

<button>begin</button>