Reliable way to determine variable message length when receiving from socket

Question

I have some Python code for capturing images from a camera and sending them to a C# server.

When sending the messages from the client- I precede the data with the message size so I know how much data to pull from the socket server-side.

It seems to work well most of the time, but occasionally - the message doesn't appear to start with the message size.

I'm not sure why this is happening but I can't figure out how to deal with it.

Python code:

while True:
    send_message("SEND_FRAME_DATA_HERE")

def send_message(message):

    message_size = len(message.encode())

    print (f"Message: {message_size} - {message}")

    my_socket.sendall(struct.pack(">L", message_size) + message.encode())

C#

private const int MESSAGE_CHUNK_SIZE = 4096;
private const int MESSAGE_PREFIX_SIZE = 4;

private void _receiveMessage(IAsyncResult ar)
{
    StateObject state = (StateObject)ar.AsyncState;
    Socket handler = state.workSocket;
    List<byte> messageBuffer = new List<byte>();
    byte[] tempBuffer = new byte[MESSAGE_CHUNK_SIZE];

    try
    {
        handler.EndReceive(ar);
        messageBuffer.AddRange(state.messageBuffer);

        while (true)
        {
            while (messageBuffer.Count < MESSAGE_PREFIX_SIZE)
            {
                handler.Receive(tempBuffer, 0, MESSAGE_CHUNK_SIZE, 0);
                messageBuffer.AddRange(tempBuffer);
            }

            int messageLength = _getMessageLength(messageBuffer);

            // Occasionally the four bytes determining message length
            // are read from what appears to be mid message
            if (messageLength > 20)
            {
                Console.Write("halp");
            }

            messageBuffer = messageBuffer.Skip(MESSAGE_PREFIX_SIZE).ToList();

            while (messageBuffer.Count < messageLength)
            {
                handler.Receive(tempBuffer, 0, StateObject.messageChunkSize, 0);
                messageBuffer.AddRange(tempBuffer);
            }

            var wholeMessage = messageBuffer.Take(messageLength).ToList();
            var messageString = Encoding.Default.GetString(wholeMessage.ToArray());

            Console.WriteLine(messageString);

            messageBuffer = messageBuffer.Skip(messageLength).ToList();
        }
    }
    catch (SocketException ex)
    {
        Console.WriteLine(ex.Message);
    }
}

private int _getMessageLength(List<byte> message)
{
    byte[] bytes = { message[3], message[2], message[1], message[0] };
    return BitConverter.ToInt32(bytes, 0);
}

The message buffer should look something like this:

On a good run:

On a bad run:

Answer 1

The problem appears to be with this code:

handler.Receive(tempBuffer, 0, StateObject.messageChunkSize, 0);
messageBuffer.AddRange(tempBuffer);

Socket.Receive() returns the number of bytes actually read into the tempBuffer. You need to save that value, and then use it to copy the correct number of bytes to messageBuffer.

int bytesRead = handler.Receive(tempBuffer, 0, StateObject.messageChunkSize, 0);
messageBuffer.AddRange(tempBuffer.Take(bytesRead));