Why is libao silent when loaded using dlopen?

Question

I'm writing an application using libao for audio output. The portion of my program that calls into libao lives in a shared object:

// playao.c
// compile with: gcc -shared -o libplayao.so playao.c -lao -lm
#include <ao/ao.h>
#include <stdio.h>
#include <math.h>

void playao(void) {
    int i;
    unsigned char samps[8000];
    ao_initialize();
    ao_sample_format sf;
    sf.bits = 8;
    sf.rate = 8000;
    sf.channels = 1;
    sf.byte_format = AO_FMT_NATIVE;
    sf.matrix = "M";
    ao_device *device = ao_open_live(ao_default_driver_id(), &sf, NULL);
    if(!device) {
        puts("ao_open_live error");
        ao_shutdown();
        return;
    }
    for(i = 0; i < 8000; ++i) {
        float time = (float)i / 8000;
        float freq = 440;
        float angle = time * freq * M_PI * 2;
        float value = sinf(angle);
        samps[i] = (unsigned char)(value * 127 + 127);
    }
    if(!ao_play(device, (char *)samps, 8000)) {
        puts("ao_play error");
    }
    ao_close(device);
    ao_shutdown();
}

If I link against this shared object in a program, it works fine:

// directlink.c
// compile with: gcc -o directlink directlink.c libplayao.so -Wl,-rpath,'$ORIGIN'
void playao(void);

int main(int argc, char **argv) {
    playao();
    return 0;
}

However, if I use dlopen/dlsym to invoke it, there are no errors, but the program does not cause any sound to be emitted:

// usedl.c
// compile with: gcc -o usedl usedl.c -ldl
#include <dlfcn.h>
#include <stdio.h>

int main(int argc, char **argv) {
    void *handle = dlopen("./libplayao.so", RTLD_LAZY);
    if(!handle) {
        puts("dlopen failed");
        return 1;
    }
    void *playao = dlsym(handle, "playao");
    if(!playao) {
        puts("dlsym failed");
        dlclose(handle);
        return 1;
    }
    ((void (*)(void))playao)();
    dlclose(handle);
    return 0;
}

However, running usedl with LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libao.so.4 does work. So there's something about libao that wants to be loaded when the program starts up, and doesn't like being loaded any later.

Why is this? Is there any way to work around this, so that libao works correctly even if loaded later in the program's execution?

I'm running Debian 10 "buster" if it matters.

Answer 1

I asked about this on the #xiph channel on Freenode and xiphmont suggested turning turning on verbose mode. Once I did that, the failing case started getting the message:

ERROR: Failed to load plugin /usr/lib/x86_64-linux-gnu/ao/plugins-4/libalsa.so => dlopen() failed

So libao itself is trying to dlopen something, and it's failing. It's not showing me any more details, so I ran the program under GDB and set a breakpoint on dlopen. After hitting the dlopen breakpoint for libalsa and running finish, I tried finding what the error was by using print (const char *)dlerror(). And with this, I get a more detailed error:

/usr/lib/x86_64-linux-gnu/ao/plugins-4/libalsa.so: undefined symbol: ao_is_big_endian

So ao's libalsa plugin is trying to reference symbols back in libao, but it's not finding them. Why could this be? Referencing the dlopen documentation, I see:

Zero or more of the following values may also be ORed in flags:

RTLD_GLOBAL: The symbols defined by this shared object will be made available for symbol resolution of subsequently loaded shared objects.

RTLD_LOCAL: This is the converse of RTLD_GLOBAL, and the default if neither flag is specified. Symbols defined in this shared object are not made available to resolve references in subsequently loaded shared objects.

Because my dlopen call only used RTLD_LAZY and didn't include RTLD_GLOBAL or RTLD_LOCAL, it defaulted to RTLD_LOCAL, which does not expose the symbols in the shared object (like ao_is_big_endian) to subsequently loaded shared objects (like libalsa.so).

So, I tried changing the code from:

void *handle = dlopen("./libplayao.so", RTLD_LAZY);

To:

void *handle = dlopen("./libplayao.so", RTLD_LAZY | RTLD_GLOBAL);

And lo and behold, it works!