audiounity-game-engine

Realtime 3D Audio Streaming and Playback


Currently, I am working on streaming audio to Unity over network, I successfully integrated media library (GStreamer) with Unity and I was able to play the audio inside the environment using audio filter callback function attached to AudioSource:

void OnAudioFilterRead(float[] data, int channels)
{
      // fill data array with the streamed audio data
      //....
}

The previous function provided 2D audio playback with very low latency,

In my application I want to render the audio in 3D Spatial space, so the audio rendering will be dependent on camera's (Audio listener) orientation.

I tried to stream audio data into AudioClip using the following:

AudioClip TargetClip;
public AudioSource TargetSrc;

void Start()
{
    int freq=32000; //streamed audio sampling rate

    TargetClip = AudioClip.Create ("test_Clip", freq, 1, freq, true,true, OnAudioRead,OnAudioSetPosition);

    TargetSrc.clip = TargetClip;
    TargetSrc.Play ();
}

void OnAudioRead(float[] data) {
      // fill data array with the streamed audio data
      //....
}

void OnAudioSetPosition(int newPosition) {

}

When I played the audio, the audio was rendered as I wanted in 3D spatial space, however there was a huge latency (more than 2 seconds).

Is there any way to solve the latency problem?


Solution

  • I figured out how to solve this issue.

    For those facing similar problem, actually the method I was using in the first place was not efficient by filling the AudioClip data. I was supposed to use OnAudioFilterRead() in either cases, however it should be using as following for spatial audio calculations:

    public AudioSource TargetSrc;
    
    void Start()
    {
        var dummy = AudioClip.Create ("dummy", 1, 1, AudioSettings.outputSampleRate, false);
    
        dummy.SetData(new float[] { 1 }, 0);
        TargetSrc.clip = dummy; //just to let unity play the audiosource
        TargetSrc.loop = true;
        TargetSrc.spatialBlend=1;
        TargetSrc.Play ();
    }
    void OnAudioFilterRead(float[] data, int channels)
    {
        // "data" contains the weights of spatial calculations ready by unity
        // multiply "data" with streamed audio data
        for(int i=0;i<data.Length;++i)
        {
            data[i]*=internal_getSample(i);
        }
    }
    

    Hope this would help some else.