Search code examples
androidffmpegexoplayermediaextractorandroid-mediacodec

MediaExtractor for audio, getting unexpected audio


Using the MediaExtractor class, I am able to get encoded audio sample data from an saved mp4 video with the below:

ByteBuffer byteBuffer = ByteBuffer.allocate(1024 * 256);
MediaExtractor audioExtractor = new MediaExtractor();
try {
    int trackIndex = -1;
    audioExtractor.setDataSource(originalMediaItem.getFilePath());

    for (int i = 0; i < audioExtractor.getTrackCount(); i++) {
        MediaFormat format = audioExtractor.getTrackFormat(i);
        String mime = format.getString(MediaFormat.KEY_MIME);

        if (mime.startsWith("audio/")) {
            trackIndex = i;
            break;
        }
    }

    audioExtractor.selectTrack(trackIndex);

    mAudioFormatMedia = audioExtractor.getTrackFormat(trackIndex);
    mAudioTrackIndex = mMediaMuxer.addTrack(mAudioFormatMedia);

    int size = audioExtractor.readSampleData(byteBuffer, 0);
    do {
        if (audioExtractor.getSampleTrackIndex() == 1) {
            long presentationTime = audioExtractor.getSampleTime();
            mInputBufferHashMap.put(presentationTime, byteBuffer);
            audioExtractor.advance();
            size = audioExtractor.readSampleData(byteBuffer, 0);
        }
    } while (size >= 0);
    audioExtractor.release();
    audioExtractor = null;
} catch (IOException e) {
    e.printStackTrace();
}

I have a video source coming from a GlSurface and then want to use a MediaMuxer to mux this video with the audio extraction mentioned previously. Audio is interleaved into the muxer using the hashmap as video is being processed. I am successful in muxing both the Video and Audio and creating a playable mp4 video, however the audio does not sound anything like the original audio of the original mp4.

I do see the expected bufferinfo.size and bufferInfo.presentationTimeUs when I write to the muxer:

mMediaMuxer.writeSampleData(mAudioTrackIndex, buffer, mAudioBufferInfo);
Log.d(TAG, String.format("Wrote %d audio bytes at %d", mAudioBufferInfo.size, mAudioBufferInfo.presentationTimeUs));

I've tried to use the standard inputBuffer, outputBuffer with MediaCodec, like this https://gist.github.com/a-m-s/1991ab18fbcb0fcc2cf9, but this produces the same audio, and from my understanding, MediaExtractor should already be encoded audio data, so data should be able to be piped directly.

What is also interesting is that when i check for the flags when initially extracting:

if( (audioExtractor.getSampleFlags() & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) 
   Log.d(TAG, "BUFFER_FLAG_END_OF_STREAM")

Neither of the above get printed for the original mp4 video. I am now questioning the original mp4 video and whether if it is possible to have a non-extractable audiotrack for an mp4 and how I can possibly confirm this.

I believe I've looked at most if not all the MediaExtractor questions on stackoverflow and a lot of the singleton solutions for MediaExtractor on github. Does anyone know of a way to extract audio another way, i.e. using ExoPlayer (preferrably not ffmpeg because it adds a ton of overhead on the android project). Any insights would help if there are any errors in my current implementation!

EDIT 1: This is what the format is audioExtractor.getTrackFormat(trackIndex):

{max-bitrate=512000, sample-rate=48000, track-id=2, durationUs=22373187, mime=audio/mp4a-latm, profile=2, channel-count=4, language=```, aac-profile=2, bitrate=512000, max-input-size=1764, csd-0=java.nio.HeapByteBuffer[pos=0 lim=2 cap=2]}


Solution

  • Problem was attempting to create a Map for the audio data. The AudioData was not correct. I was able to solve this by batching audio sample data while writing videoData using a method like the below:

        private void writeAudioSampleData(
                    MediaExtractor audioExtractor, MediaMuxer muxer, int filterStart, int filterEnd) {
            mFilterStart = filterEnd;
            MediaCodec.BufferInfo audioBufferInfo = new MediaCodec.BufferInfo();
            boolean audioExtractorDone = false;
            audioExtractor.seekTo(filterStart, MediaExtractor.SEEK_TO_CLOSEST_SYNC);
    
            synchronized (mAudioLockObject) {
                while (!audioExtractorDone) {
                    try {
                        audioBufferInfo.size =
                                audioExtractor.readSampleData(audioInputBuffer, 0);
                    } catch (Exception e) {
                        e.printStackTrace();
                    }
    
                    if (DEBUG) {
                        Log.d(TAG, "audioBufferInfo.size: " + audioBufferInfo.size);
                    }
    
                    if (audioBufferInfo.size < 0) {
                        audioBufferInfo.size = 0;
                        audioExtractorDone = true;
                    } else {
                        audioBufferInfo.presentationTimeUs = audioExtractor.getSampleTime();
                        if (audioBufferInfo.presentationTimeUs > filterEnd) {
                            break;      //out of while
                        }
                        if (audioBufferInfo.presentationTimeUs >= filterStart &&
                                audioBufferInfo.presentationTimeUs <= filterEnd) {
                            audioBufferInfo.presentationTimeUs -= mOriginalMediaItem.mRecordingStartTs;
                            audioBufferInfo.flags = audioExtractor.getSampleFlags();
                            try {
                                muxer.writeSampleData(mAudioTrackIndex, audioInputBuffer,
                                        audioBufferInfo);
                                if (DEBUG)Log.d(TAG, String.format("Wrote %d audio bytes at %d",
                                        audioBufferInfo.size, audioBufferInfo.presentationTimeUs));
                            } catch(IllegalArgumentException | IllegalStateException |
                                    NullPointerException ignore) {}
                        }
    
                        audioExtractor.advance();
                    }
                }
            }