Search code examples
xcodemacosdrivercore-audiotensorflow-lite

Creating an audio driver for macOS with TensorFlow Lite C

Ok, so here it goes. For the past couple of days I've been struggling with this issue immensely. Basically, what I've done is I have created an audio driver for macOS that successfully routes audio to a virtual speaker (I used the sample code from Apple: Creating an Audio Server Driver Plug-In + looked at other apps that do the same thing).

Now, I'd like to add an additional "step" to this driver, adding a machine learning component that will do something with the incoming audio stream. I explored the options, and saw that the perfect candidate for the job will be the TensorFlow Lite framework (specifically its C API). After adding the framework to my Xcode project (which was a challenge in and of itself, but I managed to compile everything successfully), I am struggling with a different issue, and that is, after I link the dynamic TF library in my project (adding the libtensorflowlite_c.dylib to "Link Binary With Libraries") the driver becomes invisible in the sound preferences panel. Once I remove the libtensorflowlite_c.dylib from the Build Phases and recompile everything, it shows up with no problem, and I can route everything through it. This leads me to two questions:

  1. Is putting the ML component in the file that sets the driver even a correct strategy?

  2. And if it is a correct approach, how can I compile the driver with the TFLite library so that it shows up in the sound preferences?

Here's a small breakdown of the things that I tried, but was unsuccessful:

  1. Add the entitlements.plist file to the project with "Disable Library Validation" flag set to YES (without it, I get this error not valid for use in process: mapped file has no Team ID and is not a platform binary (signed with custom identity or adhoc?)))
  2. Embedded and signed the libtensorflowlite_c.dylib in my project + linked the Runpath and Library search paths to the library within the scope of the project, not the usr/local/lib path.

What I am basically trying to achieve, is something like the Utterly app (Link to the app: Utterly), which also routes the audio to a virtual speaker/microphone and has that additional layer of noise reduction on top of it. Playing around with the app made me wonder if the AI component is embedded within the driver, or rather outside, since the app is visible both in the tab bar and activity monitor on Mac.

Solution

  • Well, I just finished the development of an AudioServerPlug(ASP for short). Very hard for a swift developer indeed. What I do is proxy audio output to a virtual device, then My recorder app can take audio input from this virtual driver, and the users are also able to hear the sound. Finally, I made it.

    So as for your question

    1. Is putting the ML component in the file that sets the driver even a correct strategy?

    Yes. No one limits you from doing so. Except that you have to evaluate the sound latency introduced by your Machine Learning Algorithms. Give you some hints: The ASP consists of many COM methods. Process your sound(AudioBuffers actually) in COM method xxxDoIOOperation., when the operation type is:

    enter image description here

    2.And if it is a correct approach, how can I compile the driver with the TFLite library so that it shows up in the sound preferences?

    The so-called sound preferences app is MIID(Audio MIDI setup.app).
    enter image description here

    If the init lifecycle of your ASP is good. Then your ASP should show up here. For the first few days, my ASP compiles but never shows up in the MIDI. It takes me a lot of time trouble shooting.

    Here is some tips:

    • You cannot attach to the ASP in its init stage. The system launches your ASP as an independent process. But you can find its PID with this command ps -ef|grep "Core Audio", and then you can attach to this process with this PID in Xcode. This will help you a lot when debugging your xxxDoIOOperation.

    • Add more logs in every COM method. It helps you to understand the lifecycle.

    • $ sudo launchctl unload /System/Library/LaunchDaemons/com.apple.audio.coreaudiod.plist && sudo launchctl load /System/Library/LaunchDaemons/com.apple.audio.coreaudiod.plist this command helps you reload all ASPs. Then you can use the console app to check ASP's logs. This filter process:com.apple.audio.Core-Audio-Driver-Service will help you to filter the logs. enter image description here