C# Async TCP sockets: Handling buffer size and huge transfers

When using a blocking TCP socket, I don't have to specify a buffer size. For example:

using (var client = new TcpClient())
{
    client.Connect(ServerIp, ServerPort);

    using (reader = new BinaryReader(client.GetStream()))
    using (writer = new BinaryWriter(client.GetStream()))
    {
        var byteCount = reader.ReadInt32();
        reader.ReadBytes(byteCount);
    }
}

Notice how the remote host could have sent any number of bytes.

However, when using async TCP sockets, I need to create a buffer and thus hardcode a maximum size:

 var buffer = new byte[BufferSize];
 socket.BeginReceive(buffer, 0, buffer.Length, SocketFlags.None, callback, null);

I could simply set the buffer size to, say, 1024 bytes. That'll work if I only need to receive small chunks of data. But what if I need to receive a 10 MB serialized object? I could set the buffer size to 10*1024*1024... but that would waste a constant 10 MB of RAM for as long as the application is running. This is silly.

So, my question is: How can I efficiently receive big chunks of data using async TCP sockets?

Solution

Two examples are not equivalent - your blocking code assumes the remote end sends the 32-bit length of the data to follow. If the same protocol is valid for the async - just read that length (blocking or not) and then allocate the buffer and initiate the asynchronous IO.

Edit 0:

Let me also add that allocating buffers of user-entered, and especially of network-input, size is a receipt for disaster. An obvious problem is a denial-of-service attack when client requests a huge buffer and holds on to it - say sends data very slowly - and prevents other allocations and/or slows the whole system.

Common wisdom here is accepting a fixed amount of data at a time and parsing as you go. That of course affects your application-level protocol design.