AVAudioSession provides an isOtherAudioPlaying property to determine if another app is playing audio. Is there any way to determine properties of this background audio signal (the audio data itself, volume levels, etc.) through this API or is this impossible in iOS? I am open to using other tools instead of AVAudioSession if required.
This is impossible. You get no information about other processes' audio sessions. That would be a significant privacy concern. isOtherAudioPlaying is itself discouraged after iOS 8. (Apple recommends secondaryAudioShouldBeSilencedHint for the primary purpose of determining when to mute your own audio.)