I need to stream a ton of data, and performance is the top priority. gRPC seems to be doing way more than just serializing protobuf, and honestly, it feels a bit bloated for what I need. Is it anti-pattern to manually serialize/deserialize the protobuf and send the bytes over TCP? And how much of a performance boost could I expect from doing that?
Manual protobuf serialization over TCP? Totally fine. People overcomplicate this stuff.
Basically, gRPC is like bringing a tank to a go-kart race. If you just need to move some bytes fast, just do that. Serialize your protobuf, send the bytes, done.
# Dead simple
sock.send(your_message.SerializeToString())
That's it. No rocket science. You'll probably get like 30% better performance by skipping all the gRPC overhead. HTTP/2, service discovery, all that jazz - great for big distributed systems, total overkill if you're just moving data between two points. Just make sure you handle your socket connection and maybe add a little length prefix so you know exactly how many bytes to read. But seriously, it's not complicated. Want me to show you a quick example of how to do it right?
import socket
from google.protobuf import your_message_pb2
def send_protobuf(sock, message):
data = message.SerializeToString()
sock.sendall(len(data).to_bytes(4, 'big') + data)
def receive_protobuf(sock, message_class):
length = int.from_bytes(sock.recv(4), 'big')
data = sock.recv(length)
message = message_class()
message.ParseFromString(data)
return message