Realtime 3D Audio Streaming and Playback

Question

Currently, I am working on streaming audio to Unity over network, I successfully integrated media library (GStreamer) with Unity and I was able to play the audio inside the environment using audio filter callback function attached to AudioSource:

void OnAudioFilterRead(float[] data, int channels)
{
      // fill data array with the streamed audio data
      //....
}

The previous function provided 2D audio playback with very low latency,

In my application I want to render the audio in 3D Spatial space, so the audio rendering will be dependent on camera's (Audio listener) orientation.

I tried to stream audio data into AudioClip using the following:

AudioClip TargetClip;
public AudioSource TargetSrc;

void Start()
{
    int freq=32000; //streamed audio sampling rate

    TargetClip = AudioClip.Create ("test_Clip", freq, 1, freq, true,true, OnAudioRead,OnAudioSetPosition);

    TargetSrc.clip = TargetClip;
    TargetSrc.Play ();
}

void OnAudioRead(float[] data) {
      // fill data array with the streamed audio data
      //....
}

void OnAudioSetPosition(int newPosition) {

}

When I played the audio, the audio was rendered as I wanted in 3D spatial space, however there was a huge latency (more than 2 seconds).

Is there any way to solve the latency problem?

Answer 1

I figured out how to solve this issue.

For those facing similar problem, actually the method I was using in the first place was not efficient by filling the AudioClip data. I was supposed to use OnAudioFilterRead() in either cases, however it should be using as following for spatial audio calculations:

public AudioSource TargetSrc;

void Start()
{
    var dummy = AudioClip.Create ("dummy", 1, 1, AudioSettings.outputSampleRate, false);

    dummy.SetData(new float[] { 1 }, 0);
    TargetSrc.clip = dummy; //just to let unity play the audiosource
    TargetSrc.loop = true;
    TargetSrc.spatialBlend=1;
    TargetSrc.Play ();
}
void OnAudioFilterRead(float[] data, int channels)
{
    // "data" contains the weights of spatial calculations ready by unity
    // multiply "data" with streamed audio data
    for(int i=0;i<data.Length;++i)
    {
        data[i]*=internal_getSample(i);
    }
}

Hope this would help some else.