Real time pitch shifting in swift iOS

Question

I want to implement an iOS app (in swift) that changes the pitch of phones mic input and play the modified voice through speakers as the user speaks with minimum delay (a few mili seconds at most). So far I have the following code:

import UIKit
import AVFoundation

class ViewController: UIViewController {
    
    var player : AVAudioPlayerNode!
    var timePitch : AVAudioUnitTimePitch!
    var engine : AVAudioEngine!
    
    override func viewDidLoad() {
        super.viewDidLoad()
        self.timePitch = AVAudioUnitTimePitch()
        self.player = AVAudioPlayerNode()
        self.engine = AVAudioEngine()

        timePitch.pitch = -500
        timePitch.rate = 1.0
        
        engine.inputNode.installTap(onBus: 0, bufferSize: 4410, format: engine.inputNode.outputFormat(forBus: 0), block: {
            (buffer: AVAudioPCMBuffer, time: AVAudioTime) in
            self.engine.attach(self.player)
            self.engine.attach(self.timePitch)
            self.engine.connect(self.player, to: self.timePitch, format: buffer.format)
            self.engine.connect(self.timePitch, to: self.engine.mainMixerNode, format: buffer.format)
            self.player.scheduleBuffer(buffer, at: nil, options: AVAudioPlayerNodeBufferOptions.loops, completionHandler: nil)
            self.player.play()
        })
        engine.prepare()
        do {
            try engine.start()
        } catch _ {
            print("Failed to start engine!")
        }
    }
}

But as soon as I run the app I get the following errors:

2020-07-27 01:06:03.972753+0430 Pitch Shifter[37745:5251379] [plugin] AddInstanceForFactory: No factory registered for id <CFUUID 0x6000031f40e0> F8BB1C28-BAE8-11D6-9C31-00039315CD46
2020-07-27 01:06:04.152148+0430 Pitch Shifter[37745:5251392] *** Terminating app due to uncaught exception 'com.apple.coreaudio.avfaudio', reason: 'player started when in a disconnected state'
*** First throw call stack:
(
    0   CoreFoundation                      0x00007fff23c7127e __exceptionPreprocess + 350
    1   libobjc.A.dylib                     0x00007fff513fbb20 objc_exception_throw + 48
    2   CoreFoundation                      0x00007fff23c70ff8 +[NSException raise:format:arguments:] + 88
    3   AVFAudio                            0x00007fff20a7ff67 _Z19AVAE_RaiseExceptionP8NSStringz + 156
    4   AVFAudio                            0x00007fff20b13c57 _ZN21AVAudioPlayerNodeImpl9StartImplEP11AVAudioTime + 453
    5   AVFAudio                            0x00007fff20b116cc -[AVAudioPlayerNode play] + 60
    6   Pitch Shifter                       0x000000010c06a25d $s13Pitch_Shifter14ViewControllerC11viewDidLoadyyFySo16AVAudioPCMBufferC_So0H4TimeCtcfU_ + 3005
    7   Pitch Shifter                       0x000000010c06a32f $sSo16AVAudioPCMBufferCSo0A4TimeCIeggg_AbDIeyByy_TR + 95
    8   AVFAudio                            0x00007fff20aaee1b _ZN14AVAudioNodeTap10TapMessage25RealtimeMessenger_PerformEv + 1325
    9   AVFAudio                            0x00007fff20b21e4b _ZN17RealtimeMessenger23_PerformPendingMessagesEv + 59
    10  AVFAudio                            0x00007fff20b21df2 ___ZN17RealtimeMessengerC2EN10applesauce8dispatch2v15queueE_block_invoke + 97
    11  libdispatch.dylib                   0x000000010c397d48 _dispatch_client_callout + 8
    12  libdispatch.dylib                   0x000000010c39a6ba _dispatch_continuation_pop + 552
    13  libdispatch.dylib                   0x000000010c3ada0f _dispatch_source_invoke + 2205
    14  libdispatch.dylib                   0x000000010c39e40e _dispatch_lane_serial_drain + 307
    15  libdispatch.dylib                   0x000000010c39f1b5 _dispatch_lane_invoke + 476
    16  libdispatch.dylib                   0x000000010c3a928c _dispatch_root_queue_drain + 351
    17  libdispatch.dylib                   0x000000010c3a907f _dispatch_worker_thread + 278
    18  libsystem_pthread.dylib             0x00007fff52466109 _pthread_start + 148
    19  libsystem_pthread.dylib             0x00007fff52461b8b thread_start + 15
)
libc++abi.dylib: terminating with uncaught exception of type NSException

Any help to fix this is appreciated.

Edit:

I think the problem is that the installTap callback function is never called so the chain is broken.

If the approach I took to use AVFoundation's time effects to work with real time mic input is true how can this error be fixed?

Or if my approach is in the wrong direction how can I change the pitch of an input from device's mic in real time with minimum delay (a few mili-seconds delay at most)?

Answer 1

I couldn't find a proper answer to my question above and I will appreciate if someone could answer that, but for now I found AudioKit which is a great library for sound manipulation and will do the job perfectly.

This is the link to my project's source code if someone needs this in the future.

import UIKit
import AudioKit
import AudioKitUI

class ViewController: UIViewController {

    @IBOutlet weak var slider: UISlider!
    @IBOutlet weak var label: UILabel!
    @IBOutlet var audioInputPlot: AKNodeOutputPlot!
    
    var micBooster: AKBooster?
    var pitchShifter: AKPitchShifter?
    var tracker: AKFrequencyTracker!
    let mic = AKMicrophone()
    
    override func viewDidLoad() {
        super.viewDidLoad()
        AKSettings.sampleRate = AudioKit.engine.inputNode.inputFormat(forBus: 0).sampleRate
        
        let micMixer = AKMixer(mic)
        tracker = AKFrequencyTracker.init(mic)
        micBooster = AKBooster(micMixer)

        pitchShifter = AKPitchShifter(micBooster, shift: 0)

        micBooster!.gain = 5

        AudioKit.output = pitchShifter
        
        do{
            try AudioKit.start()
        } catch {
            print("error occured")
        }
    }
}