I'm looking to a design a protocol for a client-server application and need some links to some resources that may help me.
The big part is I'm trying to create my own "packet" format so I can minimize the amount of information being sent. I'm looking for some resources to dissect their protocol, but it seems some completely lack packet design, such as SMTP (which just sends strings terminated by CLRF). What are the advantages/disadvantages of using a system like SMTP over a system that uses a custom made packet? Couldn't SMTP use only a couple bytes to cover all commands through bit flags and save bandwidth/space?
Just trying to get my head around all this.
True, but SMTP wasn't particularly optimized for space, nor is it a packet-based protocol. It sits atop TCP, and uses the stream functionality of TCP. You need to decide what is desirable in your protocol: is it performance sensitive? latency? bandwidth?
Is it going to need to run as superuser? If not, you'll probably want to use UDP or TCP.
Are you going to need guarantees on delivery? If so, TCP is probably your best option, unless you are dealing with fairly extreme performance or size issues.
Few protocols these days design individual packets, though many do send very specific data structures across the wire using TCP, or, less commonly, UDP.
If you want to really optimize for space or bandwidth, consider condensing your data as much as possible into individual bits and byte, and defining and packing structures to send it across TCP. Modern network adapters are so optimized for TCP anyway, that there is often little advantage to other strategies.