I'm sending my servers microphone's audio to the browser (mostly like this post but with some modified options).
All works fine, until you head over to a mobile or safari, where it doesn't work at all. I've tried using something like howler to take care of the frontend but with not success (still works in chrome and on the computer but not on the phones Safari/Chrome/etc). <audio> ... </audio> works fine in chrome but only on the computer.
function play_audio() {
var sound = new Howl({
src: ['audio_feed'],
format: ['wav'],
html5: true,
autoplay: true
});
sound.play();
}
How does one send a wav-generated audio feed which is 'live' that works in any browser?
EDIT 230203:
I have narrowed the error down to headers (at least what I think is causing the errors).
What headers should one use to make the sound available in all browsers?
Take this simple app.py for example:
from flask import Flask, Response, render_template
import pyaudio
import time
app = Flask(__name__)
@app.route('/')
def index():
return render_template('index.html', headers={'Content-Type': 'text/html'})
def generate_wav_header(sampleRate, bitsPerSample, channels):
datasize = 2000*10**6
o = bytes("RIFF",'ascii')
o += (datasize + 36).to_bytes(4,'little')
o += bytes("WAVE",'ascii')
o += bytes("fmt ",'ascii')
o += (16).to_bytes(4,'little')
o += (1).to_bytes(2,'little')
o += (channels).to_bytes(2,'little')
o += (sampleRate).to_bytes(4,'little')
o += (sampleRate * channels * bitsPerSample // 8).to_bytes(4,'little')
o += (channels * bitsPerSample // 8).to_bytes(2,'little')
o += (bitsPerSample).to_bytes(2,'little')
o += bytes("data",'ascii')
o += (datasize).to_bytes(4,'little')
return o
def get_sound(InputAudio):
FORMAT = pyaudio.paInt16
CHANNELS = 2
CHUNK = 1024
SAMPLE_RATE = 44100
BITS_PER_SAMPLE = 16
wav_header = generate_wav_header(SAMPLE_RATE, BITS_PER_SAMPLE, CHANNELS)
stream = InputAudio.open(
format=FORMAT,
channels=CHANNELS,
rate=SAMPLE_RATE,
input=True,
input_device_index=1,
frames_per_buffer=CHUNK
)
first_run = True
while True:
if first_run:
data = wav_header + stream.read(CHUNK)
first_run = False
else:
data = stream.read(CHUNK)
yield(data)
@app.route('/audio_feed')
def audio_feed():
return Response(
get_sound(pyaudio.PyAudio()),
content_type = 'audio/wav',
)
if __name__ == '__main__':
app.run(debug=True)
With a index.html looking like this:
<html>
<head>
<title>Test audio</title>
</head>
<body>
<button onclick="play_audio()">
Play audio
</button>
<div id="audio-feed"></div>
</body>
<script>
function play_audio() {
var audio_div = document.getElementById('audio-feed');
const audio_url = "{{ url_for('audio_feed') }}"
audio_div.innerHTML = "<audio controls><source src="+audio_url+" type='audio/x-wav;codec=pcm'></audio>";
}
</script>
</html>
Fire upp the flask development server python app.py and test with chrome, if you have a microphone you will hear the input sound (headphones preferably, otherwise you'll get a sound loop). Firefox works fine too.
But If you try the same app with any browser on an iPhone you'll get no sound, and the same goes for safari on MacOS.
There's no errors and you can see that the byte stream of the audio is getting downloaded in safari, but still no sound.
What is causing this? I think I should use some kind of headers in the audio_feed response but with hours of debugging I cannot seem to find anything for this.
EDIT 230309:
@Markus is pointing out to follow RFC7233 HTTP Range Request. And that's probably it. While firefox, chrome and probably more browsers on desktop send byte=0- as header request, safari and browsers used on iOS send byte=0-1 as header request.
EDITED 2023-03-12
It turns out that it is sufficient to convert the audio live stream to mp3. For this you can use ffmpeg. The executable has to be available in the execution path of the server process. Here is a working draft tested with windows laptop as server and Safari on iPad as client:
from subprocess import Popen, PIPE
from threading import Thread
from flask import Flask, Response, render_template
import pyaudio
FORMAT = pyaudio.paFloat32
CHANNELS = 1
CHUNK_SIZE = 4096
SAMPLE_RATE = 44100
BITS_PER_SAMPLE = 16
app = Flask(__name__)
@app.route('/')
def index():
return render_template('index.html', headers={'Content-Type': 'text/html'})
def read_audio(inp, audio):
while True:
inp.write(audio.read(num_frames=CHUNK_SIZE))
def response():
a = pyaudio.PyAudio().open(
format=FORMAT,
channels=CHANNELS,
rate=SAMPLE_RATE,
input=True,
input_device_index=1,
frames_per_buffer=CHUNK_SIZE
)
c = f'ffmpeg -f f32le -acodec pcm_f32le -ar {SAMPLE_RATE} -ac {CHANNELS} -i pipe: -f mp3 pipe:'
p = Popen(c.split(), stdin=PIPE, stdout=PIPE)
Thread(target=read_audio, args=(p.stdin, a), daemon=True).start()
while True:
yield p.stdout.readline()
@app.route('/audio_feed', methods=['GET'])
def audio_feed():
return Response(
response(),
headers={
# NOTE: Ensure stream is not cached.
'Cache-Control': 'no-cache, no-store, must-revalidate',
'Pragma': 'no-cache',
'Expires': '0',
},
mimetype='audio/mpeg')
if __name__ == "__main__":
app.run(host='0.0.0.0')
In index.html change the type to audio/mp3:
<!DOCTYPE html>
<html>
<head>
<title>Test audio</title>
</head>
<body>
<button onclick="play_audio()">
Play audio
</button>
<div id="audio-feed"></div>
</body>
<script>
function play_audio() {
var audio_div = document.getElementById('audio-feed');
const audio_url = "{{ url_for('audio_feed') }}"
audio_div.innerHTML = "<audio preload='all' controls><source src=" + audio_url + " type='audio/mp3'></audio>";
}
</script>
</html>
Disclaimer: This is just a basic demo. It opens an audio-ffmpeg subprocess for each call to the audio_feed handler. It doesn't cache data for multiple requests, it doesn't remove unused threads and it doesn't delete data that isn't consumed.