Python TCP Socket losing data in recv [acting weird]

Question

I wrote a simplistic socket client for reading data in Python 3.4

The problem I'm having is that when the server sends a small amount of data (around 1000) bytes, it will read it perfectly but when a large chunk of data is being handled around (9500 bytes) it will only give me a small chunk of data (like 1100-ish chunks). I can't seem to figure out why its behaving so erratically when handling the huge amount of data. I know that my data is not larger than ssize_t maximum of 32767.

It works perfectly when handling small data and completely turns 180 and behaves differently when handling a huge amount of data. I know that this is not a problem in the TCP server because I tested it with a PHP TCP client and it worked perfectly when handling the huge amount of data.

Any help is greatly appreciated.

import socket
import json

# Written in Python 3.4.

class SocketClient:

    def __init__(self, host, port, format, timeout = None):
        # Constructor
        self.host = host
        self.port = port
        self.format = format
        self.timeout = timeout

        self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    def send(self, firstname, lastname, parameters = [], format = 'json'):
        if self.socket is not None:
            self.socket.connect((self.host, self.port))

            data = {}
            data['firstname'] = firstname
            data['lastname'] = lastname
            data['parameters'] = parameters
            data['format'] = format

            self.socket.send(bytes(json.dumps(data), "utf-8"))

            result = self.socket.recv(32767)
            result = result.decode()

            return result

    def shutdown(self):
        if socket is not None:
            self.socket.shutdown(socket.SHUT_RDWR)
            self.socket.close()


if __name__ == __main__:
    client = SocketClient("127.0.0.1", 8080, 'json')
    response = client.send('foo', 'bar', ['foobar'])
    print(response)
    client.shutdown()

Answer 1

TCP is a streaming protocol. Data is delivered in junks of bytes, where the length is determined by many factors. One is, that internal buffers are limited to some thousand bytes. You never can read 32767 bytes at once. The only guarantee with recv is, that you get at least 1 byte and at most the number of bytes you say. Your code have to cope with this, that means, you have to do more than one recv call until you have the amount of bytes you need. That means on the other side, a protocol that don't have end-of-message indicators or a length encoded is badly broken. In your case: you have to parse the json byte stream until a valid json expression is sent. But what about 1245.6? Is it finished after receiving 1 or 12 or ...? To repair your protocol, simply send some length information with your json data.

For sending you should use sendall instead of send.