I'm processing about 500,000 audio files, each about 10 seconds long. The bottleneck is reading the audio files. I'm currently using pydub, and have tried reading both as mp3 (original source) and wav (after ffmpeg conversion). Both are too slow, and would take more than 3 days!
Does anyone know if there's a faster method I can use to read these audio files? Thanks!
audiofile is focused on reading speed is among the fastest libraries for Python, according to this benchmark. The benchmark might not be independent, but it's not much biased either. It's a solid choice.
Assuming you have fast drive, you should use audiofile in tandem with joblib for parallel read (not multiprocessing). The number of jobs you can fire should depend on when the system becomes either I/O- (more likely) or CPU-bound.
If you'd like to stick to ffmpeg, consider parallel conversion with GNU Parallel:
OUTPUT_DIR="YOUR_DIRECTORY"
find . -name '*.mp3' -type f -print0 | parallel -0 ffmpeg -f s16le -ar 48000 -ac 1 -i {} -filter:a loudnorm ${OUTPUT_DIR}/{.}.wav
Parameters:
-f s16le - signed 16-bit little endian samples-ar 48000 - sample rate 48kHz-ac 1 - 1 channel (mono)-i file.raw - input file-filter:a loudnorm EBU R128 loudness normalization