NKE. Can't handle file uploads and other high load connections

Question

I got a problem when using a kernel extension that filters network traffic. My code was written according to Apple's tcplognke example.

Everything goes OK but when I attempt to upload a file bigger than 500 kb - connection drops.

Here is simplified kext code:

errno_t tl_data_fn(void *cookie, socket_t so, const struct sockaddr *addr, mbuf_t *data, mbuf_t *control, sflt_data_flag_t flags, FilterSocketDataDirection direction) {
    errno_t result = 0;

    if (check_tag(data, gidtag, FILTER_TAG_TYPE, direction == FilterSocketDataDirectionIn ? IN_DONE : OUT_DONE)) {
        return result;
    }

    if (!cookie) return result;

    filter_cookie *f_cookie = get_filter_cookie(cookie);

    uint32_t data_size = (uint32_t)mbuf_pkthdr_len(*data);
    uint32_t offset = 0;

    printf("tl_data_ft: %d", data_size);

    while (offset < data_size) {
        FilterNotification notification;

        if (direction == FilterSocketDataDirectionIn) {
            notification.event = FilterEventDataIn;
        } else {
            notification.event = FilterEventDataOut;
        }
        notification.socketId = (uint64_t)so;
        notification.inputoutput.dataSize = min(data_size - offset, sizeof(notification.inputoutput.data));

        mbuf_copydata(*data, offset, notification.inputoutput.dataSize, notification.inputoutput.data);
        offset += notification.inputoutput.dataSize;

        send_notification(f_cookie, &notification);
    }

    result = EJUSTRETURN;

    if (result == EJUSTRETURN) {
        mbuf_freem(*data);

        if (control != NULL && *control != NULL)
            mbuf_freem(*control);
    }

    return result;
}

errno_t tl_data_in_fn(void *cookie, socket_t so, const struct sockaddr *from, mbuf_t *data, mbuf_t *control, sflt_data_flag_t flags) {
    return tl_data_fn(cookie, so, from, data, control, flags, FilterSocketDataDirectionIn);
}

errno_t tl_data_out_fn(void *cookie, socket_t so, const struct sockaddr *to, mbuf_t *data, mbuf_t *control, sflt_data_flag_t flags) {
    return tl_data_fn(cookie, so, to, data, control, flags, FilterSocketDataDirectionOut);
}

And the user space code:

int s = socket(PF_SYSTEM, SOCK_DGRAM, SYSPROTO_CONTROL);  

//connect to driver  

FilterNotification notification;  
while (recv(s, &notification, sizeof(FilterNotification), 0) == sizeof(FilterNotification)) {  
    FilterClientResponse response;  
    response.socketId = notification.socketId;  
    response.direction = (notification.event == FilterEventDataIn) ? FilterSocketDataDirectionIn : FilterSocketDataDirectionOut;  
    response.dataSize = notification.inputoutput.dataSize;  
    memcpy(response.data, notification.inputoutput.data, notification.inputoutput.dataSize);  
    send(s, &response, sizeof(response), 0);  
}  

When I asked on apple develper forum, the developer said "I don’t see any attempt to handle send-side flow control here. Without that a file upload can easily eat up all of the available mbufs, and things will go badly from there" but there is not examples at all. Can someone help me? thanks.

Answer 1

The problem was in socket buffer. When I inject a lot of data very quickly, the buffer becomes full and inject_data_in/inject_data_out functions returns error. The workaround is to store pending packets in kernel space (You can use TAILQ for example) and then, when socket becomes available for writing (To get this event you can use kqueue on OS X) continue injection