Search code examples
pythonsocketstcp

Socket Programming Python: How to make sure entire message is received?


I am using python 3.x and the socket module. The server is running on an ipv4 address and using tcp. I read some tutorials on how to send and receive data. For the server or client to make sure the entire message was sent you can simply check if the amount of sent data is equals the size of the message:

def mysend(self, msg):
    totalsent = 0
    while totalsent < MSGLEN:
        sent = self.sock.send(msg[totalsent:])
        if sent == 0:
            raise RuntimeError("socket connection broken")
        totalsent = totalsent + sent

Source: https://docs.python.org/3/howto/sockets.html#socket-howto

And for the client to make sure the entire response has been received this tutorial recommends to add the size of the response at the beginning of the response.

My questions:

  1. How can I make sure I receive the first part of the message indicating the size of the message (assuming my message contains 1000 characters I would need four characters to indicate the size)?
  2. Why can't I just add a specified symbol like '<' at the begging of the message and '>' at the end so I know where it start and ends?

Edit:

  1. When I use sock.recv(1024) and my messages just have a size of 500 to 1000 characters doesn't that make sure I receive all of them?

Solution

  • First of all, to send all the bytes you don't need a loop because python sockets provide a simple method: socket.sendall().

    Now to your questions:

    1. Yes, even to receive just 4 bytes you should have a receive loop that calls recv() on the socket until 4 bytes are read.

    2. You can, if you can guarantee that such characters will not appear in the message itself. However, you'd still need to search every character that you read in for the magic delimiter, so it seems inferior to simply prefixing the message body with a length.

    3. When you call recv(n) that is only guaranteed to return at most n bytes, not exactly n bytes.

    Here are three different recvall() methods to compare:

    def recvall(sock, size):
        received_chunks = []
        buf_size = 4096
        remaining = size
        while remaining > 0:
            received = sock.recv(min(remaining, buf_size))
            if not received:
                raise Exception('unexpected EOF')
            received_chunks.append(received)
            remaining -= len(received)
        return b''.join(received_chunks)
    

    and the much shorter

    def recvall2(sock, size):
        return sock.recv(size, socket.MSG_WAITALL)
    

    and finally another version that is a little shorter than the first but lacks a couple of features:

    def recvall3(sock, size):
        result = b''
        remaining = size
        while remaining > 0:
            data = sock.recv(remaining)
            result += data
            remaining -= len(data)
        return result
    

    The second one is nice and short, but it relies on a socket option socket.MSG_WAITALL that I do not believe is guaranteed to exist on every platform. The first and third ones should work everywhere. I haven't really benchmarked any to compare and contrast.