Delay/latency in synchronous boost asio with unix socket

Question

I write a client-server app which uses asynchronous boost asio networking (boost::asio::async_write and boost::asio::async_read) on server side and synchronous calls (boost::asio::write and boost::asio::read) on the client end. Because underneath I use protocol buffers, if I want to send a buffer from the client, first I send the payload size, then in the second call the payload body. Pseudocode for the client end:

void WriteProtobuf( std::string && body )
{
  boost::system::error_code ec;
  std::size_t dataSize = body.size();
  // send the size
  boost::asio::write( socket, boost::asio::buffer( reinterpret_cast<const char *>( &dataSize ), sizeof( dataSize ) ), ec );
  // send the body
  boost::asio::write( socket, boost::asio::buffer( body.data(), body.size() ), ec );
}

Pseudocode for the server end:

void ReadProtobuf()
{
  std::size_t requestSize;
  std::string body;
  // read the size
  boost::asio::async_read( socket, boost::asio::buffer( &requestSize, sizeof( requestSize ) ), [&requestSize, &body]() { // read the size
    body.resize( requestSize );
    // read the body
    boost::asio::async_read( socket, boost::asio::buffer( body.data(), body.size() ), []() {
        /* ... */
    });
  });
}

Now, it works just fine, but I observe a ~40ms latency in the second boost::asio:write call. I found an easy but not clean solution to work it around. I added the "confirmation" byte send from the server between the calls of write from client:

Pseudocode for the client end:

void WriteProtobuf( std::string && body )
{
  boost::system::error_code ec;
  std::size_t dataSize = body.size();
  // send the size
  boost::asio::write( socket, boost::asio::buffer( reinterpret_cast<const 
char *>( &dataSize ), sizeof( dataSize ) ), ec );
  char ackByte;
  // read the ack byte
  boost::asio::read( socket, boost::asio::buffer( ackByte, sizeof( ackByte ) ), ec );
  // send the body
  boost::asio::write( socket, boost::asio::buffer( body.data(), body.size() ), ec );
}

Pseudocode for the server end:

void ReadProtobuf()
{
  std::size_t requestSize;
  std::string body;
  // read the size
  boost::asio::async_read( socket, boost::asio::buffer( &requestSize, sizeof( requestSize ) ), [&requestSize, &body]() { // read the size
    body.resize( requestSize );
    char ackByte = 0;
    // write the ack byte
    boost::asio::async_write( socket, boost::asio::buffer( &ackByte, sizeof( ackByte ), []() {
        // read the body
        boost::asio::async_read( socket, boost::asio::buffer( body.data(), body.size() ), []() {
            /* ... */
        });
    });
  });
}

This removes the latency but still I would get rid of unnecessary communication and understand better why is it happening this way.

Answer 1

On the other hand glueing size at the beginning of the data isn’t an option, because then I would do a copy.

Scatter-gather to the rescue: https://www.boost.org/doc/libs/1_75_0/doc/html/boost_asio/reference/buffer.html#boost_asio.reference.buffer.buffers_and_scatter_gather_i_o

So, this could help:

void WriteProtobuf(std::string const& body) {
    std::size_t dataSize = body.size();
    std::vector<asio::const_buffer> bufs {
        asio::buffer(&dataSize, sizeof(dataSize)),
        asio::buffer(body.data(), body.size())
    };

    boost::system::error_code ec;
    write(socket, asio::buffer(bufs), ec);
}

Use Protobuf

However, since you are using Protobuf, consider not serializing to a string, but using the builtin support for size-prefixed stream serialization:

void WriteProtobuf(::google::protobuf::Message const& msg) {
    std::string buf;
    google::protobuf::io::StringOutputStream sos(&buf);
    msg.SerializeToZeroCopyStream(&sos);

    boost::system::error_code ec;
    write(socket, asio::buffer(buf), ec);
}

On the receiving end you can then use the streams to read until the message is complete. See e.g. https://developers.google.com/protocol-buffers/docs/reference/csharp/class/google/protobuf/coded-input-stream

Other Considerations

If this doesn't actually help, then you could look into explicitly flushing on the socket file descriptor:

https://stackoverflow.com/a/855597/85371

So, e.g.

    ::fsync(socket.native_handle());