Search code examples
clinuxsocketssendmsgrecvmsg

Passing multiple buffers with iovec in C Linux sockets


I'm writing a linux C client-server programs that communicates with each other with unix domain sockets and passes couple of buffers each time. I'm using ioverctors but for some reasons the server program only receives the first io vector. Any idea ? I attached the relevant code snippets.

Client code:

    struct iovec iov[2];
    struct msghdr mh;
    int rc;
    char str1[] = "abc";
    char str2[] = "1234";
    iov[0].iov_base = (caddr_t)str1;
    iov[0].iov_len = sizeof(str1);
    iov[1].iov_base = (caddr_t)str2;
    iov[1].iov_len = sizeof(str2);
 
    memset(&mh, 0, sizeof(mh));
    mh.msg_iov = iov;
    mh.msg_iovlen = 2;
 
    n = sendmsg(sockfd, &mh, 0);            /* no flags used*/
    if (n > 0) {
        printf("Sendmsg successfully executed\n");
    } 
}

Server code:
{
    struct sockaddr_un *client_sockaddr = (sockaddr_un *)opq;
    struct  msghdr msg;
    struct  iovec io[2];
    char    buf[16];
    char    buf2[16];

    io[0].iov_base      = buf;
    io[0].iov_len       = sizeof(buf);
    io[1].iov_base      = buf2;
    io[1].iov_len       = sizeof(buf2);

    msg.msg_iov     = io;
    msg.msg_iovlen  = 2;
 
    int len = recvmsg(sock, &msg, 0); 

    if (len > 0) {
        printf("recv: %s %d %s %d\n", msg.msg_iov[0].iov_base, msg.msg_iov[0].iov_len, msg.msg_iov[1].iov_base, msg.msg_iov[1].iov_len);
    }
    return 0;
}

The output i'm getting from the server: recv: abc 16 16


Solution

  • sendmsg(), writev(), pwritev(), and pwritev2() do not operate on multiple buffers, but one discontiguous buffer. They operate exactly as if you'd allocate a large enough temporary buffer, gather the data there, and then do the corresponding syscall on the single temporary buffer.

    Their counterparts recvmsg(), readv(), preadv(), and preadv2() similarly do not operate on multiple buffers, only on one discontiguous buffer. They operate exactly as if you'd allocate a large enough temporary buffer, receive data into that buffer, then scatter the data from that buffer to the discontiguous buffer parts.

    Unix domain datagram (SOCK_DGRAM) and seqpacket (SOCK_SEQPACKET) sockets preserve message boundaries, but stream sockets (SOCK_STREAM) do not. That is, using a datagram or seqpacket socket you receive each message as it was sent. With a stream socket, message boundaries are lost: two consecutively sent messages can be received as a single message, and you can (at least in theory) receive a partial message now and the rest later.

    You can use the Linux-specific sendmmsg() function to send several messages in one call (using the same socket). If you use an Unix domain datagram or seqpacket socket, these will then retain their message boundaries.

    Each message is described using a struct mmsghdr. It contains struct msghdr msg_hdr; and unsigned int msg_len;. msg_hdr is the same as you use when sending a single message using e.g. sendmsg(); you can use more than one iovec for each message, but the recipient will receive them concatenated into a single buffer (but can scatter that buffer using e.g. recvmsg()). msg_len will be filled in by the sendmmsg() call: the number of bytes sent for that particular message, similar to the return value of e.g. sendmsg() call when no errors occur.

    The return value from the sendmmsg() call is the number of messages sent successfully (which may be fewer than requested!), or -1 if an error occurs (with errno indicating the error as usual). Thus, you'll want to write a helper function or a loop around sendmmsg() to make sure you send all the messages. For portability, I recommend a helper function, because you can then provide another based on a loop around sendmsg() for use when sendmmsg() is not available.

    The only real benefit of sendmmsg() is that you need fewer syscalls to send a large number of messages: it boosts efficiency in certain situations, that's all.