Search code examples
javascriptnode.jsffmpeggoogle-speech-apiweb-audio-api

Sending 2 streams to FFmpeg from nodejs

I am trying to send 2 ReadableStreams to FFmpeg from nodejs. I have tried using fluent-ffmpeg library to do this, but it only supports sending one stream for processing. Check here

My problem is: I have 2 incoming mono audio streams, I want to send them to ffmpeg to create a stereo stream, which I will then send to google's speech to text service, to generate a transcription.

I am successfully receiving both the mono streams to the nodejs server. How to utilize FFmpeg to merge them in realtime is still unclear, I could spawn a FFmpeg child process, but I'm not sure how to give 2 ReadableStreams as inputs and get the output as another stream? FFmpeg supports multiple input streams.

I can merge the 2 mono streams if they are in two separate files with this code. 

const { spawn } = childProcess;
const ffmpeg = spawn('ffmpeg', [
  '-i', this.phoneAudioFile,
  '-i', this.micAudioFile,
  '-filter_complex', '[0:a][1:a]amerge=inputs=2[a]',
  '-map', '[a]',
  this.outputLosslessFile,
]);

How can I acheive the same using 2 streams instead of 2 files?

EDIT

  • The incoming streams both have PCM audio data.
  • This entire process runs on a linux Ubuntu server.
  • The final output must be a wav file.
Solution

  • Assuming your source audio streams are regular PCM audio (such as what is most commonly found in WAV files), I would merge the streams internally in your application, and output a single stream to FFmpeg.

    This can be done as simply as alternating which stream you read from, effectively interleaving the samples.

    If your samples are 16-bit, then each sample is two bytes. So, your stream will look like this:

    [LL][RR][LL][RR][LL][RR]

    (where each LL is 2 bytes of a single sample for the left channel, and the same for RR)

    If you're going to pipe this into FFmpeg, you'll need to set up the appropriate parameters for RAW PCM. Or, you can generate a WAV file header in your application as well.