Playing audio node. play.ai

Question

Based on the details from play.ai's docs, it should be easy to play base64 audio but I'm running into issues. I am using nuxtjs.

These are the docs

myWs.on('message', (message) => {
  const event = JSON.parse(message);
  if (event.type === 'audioStream') {
    // deserialize event.data from a base64 string to binary
    // enqueue/play the binary data at your player
    return;
  }
});

How do I play audio from this.

Answer 1

This answer is for others trying to do this. This is the current iteration.

ws.onopen = () => {
  console.info('connected')
  ws.send(
    JSON.stringify({
      type: 'setup',
      apiKey: config.public.playai.apiKey,
      outputFormat: 'mp3',
    }),
  )
}


const audioData: string[] = []
let currentChunk = ''
ws.onmessage = (message) => {
  const event = JSON.parse(message.data)

  if (event.data) {
    currentChunk += event.data
    if (event.data[event.data.length - 1] === '=') {
      audioData.push(currentChunk)
      currentChunk = ''
    }
    const lastTenCharacters = event.data.slice(event.data.length - 10)

    if (
      lastTenCharacters
        .split('')
        .every((char: string) => char === lastTenCharacters[0] && char !== 'A')
    ) {
      audioData.push(currentChunk)
      currentChunk = ''
      playAudioRecursively(audioData)
    }
  }
}

async function playAudioRecursively(audioData: string[], index = 1) {
  if (index >= audioData.length) return

  const audioBlob = (await base64ToBlob(audioData[index], 'audio/mpeg')) as Blob
  const audioUrl = URL.createObjectURL(audioBlob)
  const audio = new Audio(audioUrl)

  audio.play()
  audio.onended = () => {
    playAudioRecursively(audioData, index + 1)
  }
}

function base64ToBlob(base64: string, mimeType: string) {
  return new Promise((resolve, reject) => {
    try {
      const byteCharacters = atob(base64)
      const byteNumbers = new Array(byteCharacters.length)
      for (let i = 0; i < byteCharacters.length; i++) {
        byteNumbers[i] = byteCharacters.charCodeAt(i)
      }
      const byteArray = new Uint8Array(byteNumbers)
      resolve(new Blob([byteArray], { type: mimeType }))
    } catch (error) {
      reject(error)
    }
  })
}

Notes:

• the first iteration of audioData doesn't work that's why you have to start from index = 1 and not index = 0 when playing the audio
• because we are dealing with a stream of base64 data, you have to check where the "=" are and end the chunk. In base64, "=" is padding basically saying this is the end of the part
• In my current understanding, when you see the last 10 characters all = "V" or "q" it means the end of transmission.

Please feel free to add comments below as I don't fully understand how audio mp3 streams work. These are just the patterns I picked up to get this working.