Playing audio while applying video filter in GPUImage

Question

I've googled this question but did not find any solution.

So, my problem is, I'm applying video filter using GPUImage to a video file. At same time, sound of that video is not playing. I know that sound playing is not supported in GPUImage while applying filters.

So, how can I achieve that?

Answer 1

You can achieve sound playing by adding support for AVPlayer like this.

1. In GPUImageMovie.h,

   @property(readwrite, nonatomic) BOOL playSound;

2. In GPUImageMovie.m, update @interface GPUImageMovie () like this.

   @interface GPUImageMovie() <AVPlayerItemOutputPullDelegate> 
   {
       // Add this
       BOOL hasAudioTrack;
   
       .........
       ........
   
      // Add all below three lines
      AVPlayer *theAudioPlayer;
      CFAbsoluteTime startActualFrameTime;
      CGFloat currentVideoTime;
   }
   

3. After that, in -(void)startProcessing method, add following lines.

   - (void)startProcessing
   {
      // Add this
      currentVideoTime = 0.0f;
   
      .....
      .....
   
      // Add this
      if (self.playSound)
      {
          [self setupSound];
      }
   
      GPUImageMovie __block *blockSelf = self;
   
      [inputAsset loadValuesAsynchronouslyForKeys:[NSArray arrayWithObject:@"tracks"] completionHandler: ^{
       runSynchronouslyOnVideoProcessingQueue(^{
   
               .........
               .........
   
               // Add this
               startActualFrameTime = CFAbsoluteTimeGetCurrent() - currentVideoTime;
   
               .......
               .......
           });
      }];
   }
   

4. Again, In -(AVAssetReader*)createAssetReader update it,

   - (AVAssetReader*)createAssetReader
   {
       .......
       .......  
       BOOL shouldRecordAudioTrack = (([audioTracks count] > 0) &&  (self.audioEncodingTarget != nil));
   
       // Add this
       hasAudioTrack = ([audioTracks count] > 0);
   
       .........
   }
   

5. In - (void)processAsset,

   - (void)processAsset
   {
   
      .......
      .......
   
      if ([reader startReading] == NO)
      {
         NSLog(@"Error reading from file at URL: %@", self.url);
         return;
      }
   
      // Add this
      if (self.playSound && hasAudioTrack)
      {
         [theAudioPlayer seekToTime:kCMTimeZero];
         [theAudioPlayer play];
      }
   
      .........
      .........
   }
   

6. In -(void)endProcessing,

   - (void)endProcessing
   {
      .........
      .........
   
      if (self.playerItem && (displayLink != nil))
      {
         [displayLink invalidate]; // remove from all run loops
         displayLink = nil;
      } 
   
      // Add this
      if (theAudioPlayer != nil)
      {
         [theAudioPlayer pause];
         [theAudioPlayer seekToTime:kCMTimeZero];
      }
   
      .........
      .........
   }
   

7. Add new method -(void)setupSound at last,

   // Add this
   - (void)setupSound
   {
      if (theAudioPlayer != nil)
      {
          [theAudioPlayer pause];
          [theAudioPlayer seekToTime:kCMTimeZero];
          theAudioPlayer = nil;
      }
   
      theAudioPlayer = [[AVPlayer alloc] initWithURL:self.url];
   }
   

8. At last, when you create your GPUImageMovie object, don't forget to set playSound flag to YES.

   Like this : movieFile.playSound = YES;

   Hope this helps.