lio_listio: How to wait until all requests complete?

In my C++ program, i use the lio_listio call to send many (up to a few hundred) write requests at once. After that, I do some calculations, and when I'm done I need to wait for all outstanding requests to finish before I can submit the next batch of requests. How can I do this?

Right now, I am just calling aio_suspend in a loop, with one request per call, but this seems ugly. It looks like I should use the struct sigevent *sevp argument to lio_listio. My current guess is that I should do something like this:

In the main thread, create a mutex and lock it just before the call to lio_listio.
In the call to lio_listio, specify a notification function / signal handler that unlocks this mutex.

This should give me the desired behavior, but will it work reliably? Is it allowed to manipulate mutexes from the signal handler context? I read that pthread mutexes can provide error detection and fail with if you try to lock them again from the same thread or unlock them from a different thread, yet this solution relies on deadlocking.

Example code, using a signal handler:

void notify(int, siginfo_t *info, void *) {
    pthread_mutex_unlock((pthread_mutex_t *) info->si_value);
}

void output() {
    pthread_mutex_t iomutex = PTHREAD_MUTEX_INITIALIZER;

    struct sigaction act;
    memset(&act, 0, sizeof(struct sigaction));
    act.sa_sigaction = &notify;
    act.sa_flags = SA_SIGINFO;
    sigaction(SIGUSR1, &act, NULL);

    for (...) {
        pthread_mutex_lock(&iomutex);

        // do some calculations here...

        struct aiocb *cblist[];
        int cbno;
        // set up the aio request list - omitted

        struct sigevent sev;
        memset(&sev, 0, sizeof(struct sigevent));
        sev.sigev_notify = SIGEV_SIGNAL;
        sev.sigev_signo = SIGUSR1;
        sev.sigev_value.sival_ptr = &iomutex;

        lio_listio(LIO_NOWAIT, cblist, cbno, &sev);
    }

    // ensure that the last queued operation completes
    // before this function returns
    pthread_mutex_lock(&iomutex);
    pthread_mutex_unlock(&iomutex);
}

Example code, using a notification function - possibly less efficient, since an extra thread is created:

void output() {
    pthread_mutex_t iomutex = PTHREAD_MUTEX_INITIALIZER;

    for (...) {
        pthread_mutex_lock(&iomutex);

        // do some calculations here...

        struct aiocb *cblist[];
        int cbno;
        // set up the aio request list - omitted

        struct sigevent sev;
        memset(&sev, 0, sizeof(struct sigevent));
        sev.sigev_notify = SIGEV_THREAD;
        sev_sigev_notify_function = &pthread_mutex_unlock;
        sev.sigev_value.sival_ptr = &iomutex;

        lio_listio(LIO_NOWAIT, cblist, cbno, &sev);
    }

    // ensure that the last queued operation completes
    // before this function returns
    pthread_mutex_lock(&iomutex);
    pthread_mutex_unlock(&iomutex);
}

Solution

If you set the sigevent argument in the lio_listio() call, you will be notified with a signal (or function call) when all the jobs in that one particular call completes. You would still need to:

wait until you receive as many notifications as you have made lio_listio() calls, to know when they're all done.
use some safe mechanism to communicate from your signal handler to your main thread, probably via a global variable (to be portable).

If you're on linux, I would recommend tying an eventfd to your sigevent instead and wait on that. That's a lot more flexible since you don't need to involve signal handlers. On BSD (but not Mac OS), you can wait on aiocbs using kqueue and on solaris/illumos you can use a port to get notified of aiocb completions.

Here's an example of how to use eventfds on linux:

As a side note, I would use caution when issuing jobs with lio_listio. You're not guaranteed that it supports taking more than 2 jobs, and some systems have very low limits of how many you can issue at a time. Default on Mac OS for instance is 16. This limit may be defined as the AIO_LISTIO_MAX macro, but it isn't necessarily. In which case you need to call sysconf(_SC_AIO_LISTIO_MAX) (see docs). For details, see the lio_listio documentation.

You should at least check error conditions from your lio_listio() call.

Also, your solution of using a mutex is sub-optimal, since you will synchronize each loop in the for loop, and just run one at a time (unless it's a recursive mutex, but in that case its state could be corrupt if your signal handler happens to land on a different thread).

A more appropriate primitive may be a semaphore, which is released in the handler, and then (after your for loop) acquired the same number of times as you looped, calling lio_listio(). But, I would still recommend an eventfd if it's OK to be linux specific.