aio_write on Large Files

Question

I'm trying to essentially mimic the functionality of sendfile(2) in an asynchronous fashion using aio_read(3) and aio_write(3).

Everything seems to be working fine, with the exception of testing large (> 150k) files.

I have a simple struct io_request I am using to keep track of the transfers:

struct io_request {
    int status;
    struct aiocb *aiocbp;
    int sfd;
};

First, I build the aio_read() call:

struct io_request * ioreq = malloc(sizeof(struct io_request));
ioreq->status = EINPROGRESS;
ioreq->sfd = sfd;

struct aiocb * aiocbreq = malloc(sizeof(struct aiocb));
memset(aiocbreq, 0, sizeof(struct aiocb));

ioreq->aiocbp = aiocbreq;

ioreq->aiocbp->aio_fildes = ffd;

if (ioreq->aiocbp->aio_fildes == -1) {
    perror("aio_fildes");
}

ioreq->aiocbp->aio_buf = malloc(st.st_size);
if (ioreq->aiocbp->aio_buf == NULL) {
    perror("aio_buf malloc");
}

ioreq->aiocbp->aio_nbytes = st.st_size;
ioreq->aiocbp->aio_reqprio = 0;
ioreq->aiocbp->aio_offset = 0;
ioreq->aiocbp->aio_sigevent.sigev_signo = IO_READ_SIGNAL;
ioreq->aiocbp->aio_sigevent.sigev_value.sival_ptr = ioreq;
if (aio_read(ioreq->aiocbp) == -1) {
    perror("aio_read");
}

Which then later is captured in a IO_READ_SIGNAL handler:

static void
aio_read_handler(int sig, siginfo_t *si, void *ucontext)
{
    if (si->si_code == SI_ASYNCIO) {
        struct io_request *ioreq = si->si_value.sival_ptr;

        // Build the AIO write request
        struct aiocb aiocbreq;
        memset(&aiocbreq, 0, sizeof(struct aiocb));
        aiocbreq.aio_fildes = ioreq->sfd;
        aiocbreq.aio_buf = ioreq->aiocbp->aio_buf;
        aiocbreq.aio_nbytes = ioreq->aiocbp->aio_nbytes;
        aiocbreq.aio_sigevent.sigev_signo = IO_WRITE_SIGNAL;
        aiocbreq.aio_sigevent.sigev_value.sival_ptr = ioreq;

        if (aio_write((void *) &aiocbreq) == -1) {
            perror("aio_write");
        }
    }
}

I can confirm that inside the handler, even for large files, the contents of ioreq->aiocbp->aio_buf is full and complete.

Later, the aio_write() is captured in a IO_WRITE_SIGNAL handler:

static void
aio_write_handler(int sig, siginfo_t *si, void *ucontext)
{
    if (si->si_code == SI_ASYNCIO) {
        struct io_request *ioreq = si->si_value.sival_ptr;

        ssize_t bytes_written = aio_return(ioreq->aiocbp);
        printf("Wrote %zu of %zu bytes\n", bytes_written, ioreq->aiocbp->aio_nbytes);
        //free(ioreq->aiocbp);
        //free(ioreq);

        if (aio_error(ioreq->aiocbp) != 0) {
            perror("aio_write_handler");
        }
    }
}

At this point aio_write() should have been completed. I check the return values and act accordingly. Both calls report the appropriate number of bytes have been written and no errors arose during the write.

The greater application is an HTTP server. I speculate that this problem arrises because the remote client cannot read fast enough to keep up with the aio_write(). When I had a sendfile() implementation of this, I had to call sendfile() multiple times to complete the file transfer.

Several direct questions:

• Why does aio_return() and aio_error() not report any problems?
• How can I fix this behavior?
• Are there ways to buffer aio_write()? I was thinking of capping of n_bytes inside struct aiocb passed to aio_write(), and just calling aio_write() multiple times from inside aio_write_handler().

Thanks for your help!

Answer 1

If I understand correctly, you're using aio_return before aio_error. The aio_return man page says,

This function should be called only once for any given request, after aio_error(3) returns something other than EINPROGRESS.