Format wrong while recording in swift

Question

I have a Flutter app and want to record audio in the native Swift part. I use an audio unit for this. I now have the problem that I cannot process the audio data in the RenderCallback correctly. When I initialize the AudioUnitRender I get the error:

1024 frames, 4 bytes/frame, expected 4096-byte buffer; ioData.mBuffers[0].mDataByteSize=2048; kAudio_ParamError
  from AU (0x10660f5e0): auou/vpio/appl, render err: -50
AudioUnitRender failed with status: -50
throwing -1
  from AU (0x10660f5e0): auou/vpio/appl, render err: -1

But that can't actually happen, because I set the audioFormat in the init method to 2 bytes/frame. Does anyone have an idea what the problem is?

Here is the code:

startRecording function

func startRecording() {
  let audioSession = AVAudioSession.sharedInstance()
  do {
    try audioSession.setCategory(.playAndRecord, options: [.allowBluetooth])
    try audioSession.setMode(.voiceChat)
    try audioSession.setActive(true)
  } catch {
    print("Failed to set up audio session: \(error.localizedDescription)")
  }
  initAudioUnit()
  AudioOutputUnitStart(ioUnitUdp!)
}

init audio unit

func initAudioUnit() {

  var status: OSStatus

  var ioUnitDescription = AudioComponentDescription(
    componentType: kAudioUnitType_Output,
    componentSubType: kAudioUnitSubType_VoiceProcessingIO,
    componentManufacturer: kAudioUnitManufacturer_Apple,
    componentFlags: 0,
    componentFlagsMask: 0
  )
  let inputComponent = AudioComponentFindNext(nil, &ioUnitDescription)
  status = AudioComponentInstanceNew(inputComponent!, &ioUnitUdp)

  var one: UInt32 = 1
  // Enable IO for recording
  status = AudioUnitSetProperty(
    ioUnitUdp!, kAudioOutputUnitProperty_EnableIO, kAudioUnitScope_Input, kInputBus, &one,
    UInt32(MemoryLayout<UInt32>.size))

  var audioFormat = AudioStreamBasicDescription()
  memset(&audioFormat, 0, MemoryLayout<AudioStreamBasicDescription>.size)
  audioFormat.mSampleRate = 8000.0
  audioFormat.mFormatID = kAudioFormatLinearPCM
  audioFormat.mFormatFlags = kAudioFormatFlagIsSignedInteger | kAudioFormatFlagIsPacked
  audioFormat.mFramesPerPacket = 1
  audioFormat.mChannelsPerFrame = 1
  audioFormat.mBytesPerFrame = 2
  audioFormat.mBytesPerPacket = 2
  audioFormat.mBitsPerChannel = 16

  status = AudioUnitSetProperty(
    ioUnitUdp!, kAudioUnitProperty_StreamFormat, kAudioUnitScope_Input, kOutputBus, &audioFormat,
    UInt32(MemoryLayout<AudioStreamBasicDescription>.size))

  // Set input callback
  var callbackStruct1: AURenderCallbackStruct = AURenderCallbackStruct()
  callbackStruct1.inputProc = renderCallbackFunction
  callbackStruct1.inputProcRefCon = UnsafeMutableRawPointer(
    Unmanaged.passUnretained(self).toOpaque())  // self
  status = AudioUnitSetProperty(
    ioUnitUdp!, kAudioOutputUnitProperty_SetInputCallback, kAudioUnitScope_Global, kInputBus,
    &callbackStruct1, UInt32(MemoryLayout<AURenderCallbackStruct>.size))

  var bypassVP: UInt32 = 0
  status = AudioUnitSetProperty(
    ioUnitUdp!, kAUVoiceIOProperty_BypassVoiceProcessing, kAudioUnitScope_Global, kOutputBus,
    &bypassVP, UInt32(MemoryLayout<UInt32>.size))

  var vpAGC: UInt32 = 0
  status = AudioUnitSetProperty(
    ioUnitUdp!, kAUVoiceIOProperty_VoiceProcessingEnableAGC, kAudioUnitScope_Global, kOutputBus,
    &vpAGC, UInt32(MemoryLayout<UInt32>.size))

  var quality: UInt32 = 127
  status = AudioUnitSetProperty(
    ioUnitUdp!, kAUVoiceIOProperty_VoiceProcessingQuality, kAudioUnitScope_Global, kInputBus,
    &quality, UInt32(MemoryLayout<UInt32>.size))

  var unmuteOutput: UInt32 = 0
  status = AudioUnitSetProperty(
    ioUnitUdp!, kAUVoiceIOProperty_MuteOutput, kAudioUnitScope_Global, kOutputBus, &unmuteOutput,
    UInt32(MemoryLayout<UInt32>.size))

  // Initialise
  status = AudioUnitInitialize(ioUnitUdp!)
}

render callback

let renderCallbackFunction: AURenderCallback = {
  (
    inRefCon,
    ioActionFlags,
    inTimeStamp,
    inBusNumber,
    inNumberFrames,
    ioData
  ) -> OSStatus in

  let appDelegate = Unmanaged<AppDelegate>.fromOpaque(inRefCon).takeUnretainedValue()

  var bufferList = AudioBufferList(
    mNumberBuffers: 1,
    mBuffers: AudioBuffer(
      mNumberChannels: 1,
      mDataByteSize: inNumberFrames * UInt32(MemoryLayout<Int16>.size),
      mData: malloc(Int(inNumberFrames) * MemoryLayout<Int16>.size)
    )
  )

  guard let data = bufferList.mBuffers.mData else {
    return noErr
  }

  // Call AudioUnitRender to obtain recorded samples
  let status = AudioUnitRender(
    appDelegate.ioUnitUdp!, ioActionFlags, inTimeStamp, inBusNumber, inNumberFrames, &bufferList)

  if status != noErr {
    print("AudioUnitRender failed with status: \(status)")
    return status
  }

  print("data: \(bufferList)")
  // DO Stuff with data here

  free(data)

  return noErr
}

ioUnitUdp is a variable in appDelegate

Answer 1

I was able to solve it. My mistake was that I only configured IO for recording.

The playback also needs a configuration. If you don't need it, you can simply deactivate it. However, since I also needed it in my case, I also had to activate it via an AudioUnitSetProperty and then also define a rendercallback.

Configuration:

var enable: UInt32 = 1

// Enable IO for recording
status = AudioUnitSetProperty(audioUnit!, kAudioOutputUnitProperty_EnableIO, kAudioUnitScope_Input, kInputBus, &enable, UInt32(MemoryLayout<UInt32>.size))
        
// Enable IO for playback
status = AudioUnitSetProperty(audioUnit!, kAudioOutputUnitProperty_EnableIO, kAudioUnitScope_Output,kOutputBus, &enable, UInt32(MemoryLayout<UInt32>.size));

You can simply disable recording or playback with enable = 0

If playback is activated then you need a playback callback:

// Set playback callback
var playbackCallbackStruct : AURenderCallbackStruct = AURenderCallbackStruct()
playbackCallbackStruct.inputProc = playbackCallback
playbackCallbackStruct.inputProcRefCon = UnsafeMutableRawPointer(Unmanaged.passUnretained(self).toOpaque())
status = AudioUnitSetProperty(audioUnit!, kAudioUnitProperty_SetRenderCallback, kAudioUnitScope_Global, kOutputBus, &playbackCallbackStruct, UInt32(MemoryLayout<AURenderCallbackStruct>.size))

This is my callback:

let playbackCallback: AURenderCallback = { (
    inRefCon,
    ioActionFlags,
    inTimeStamp,
    inBusNumber,
    inNumberFrames,
    ioData) -> OSStatus in
        
    // fill player here
}

It is also very important that the player receives audio data with every callback. If this does not happen, it will lead to errors.