Let's say I was making a client with Adobe AIR in AS3. I know that it would be easy if one was to try to decompile it, change its code, recompile it, extend its functionality to do whatever he/she pleases, and then use it to connect to the server as if it was the original file.
The server would compare the bytes of the submitted client files to the original bytes of the client files) stored in memory on the server before the Client is served a Session ID that would be required and be valid in order to communicate with the Server
In case you were wondering, this would be done with TCP obviously. Also, the client would be installed on a machine and ran from there.
The files would be submitted on connection.
The server would measure the amount of bytes sent and compare it to the servers copy of bytes in memory I talked about earlier. Obviously if the amount of bytes is any bigger or smaller than the original client in memory on the server, the connection would be disregarded and nothing would happen.
If it's the same exact amount then it would compare if the sequence of the bytes are the exact same. If they aren't, same thing, the connection would be disregarded etc.
If all checks out the server would generate a session ID for this specific client and send it out for the client to capture and have to send with any message to the server.
If the Server doesn't receive any session ID with a client's messages, then it would disregard the message and do away with the connection.
But the hacker could just go into the client and have it write the same sequence of bytes contained in the untouched files to trick the server into believing it's authentic.
So my question is, what could I do to ensure that the sequence of bytes of the client being used to connect to the server is really the client being used to connect to the server?
It is not possible to protect the protocol completely, but it is quite possible to ensure no one in their right mind would venture to untangle the protective layers. In order to get there you need to make a few steps:
Step 1: ensure that each client->server message is absolutely globally unique. Each message should contain several parameters like:
Upon receiving a message server must validate these parameters to cut off any invalid messages - they do not come from latest handshaked genuine client.
Step 2: sign the message. MD5 or SHA1 of binary message data and some secret key. Of course there should be a secret key that is not sent along with the messages. Server (that also knows that secret key) must validate the signature. You can make it safer with two signatures: one is calculated from the standard parameters (that should include all of listed above) and the secret key, then the other one is calculated from binary representation of the message (which includes the first signature) and the other secret key.
Step 3: protect the secret key and the signing algorithms. Of course, anyone with a decompiler can pull the signing algorithm out of your app and reproduce it. You wouldn't want it. Use an obfuscator tool, like SecureSWF to protect these pieces of code that sign your messages. Don't keep the secret keys in an obvious form, construct them at the moment you need them, destroy them as soon as you don't. It might be a good idea to work with ByteArray class as it has ByteArray.clear() method that completely erases the content from the memory.
Step 4: throw some paranoia in. Make some additional checks before signing your client->server messages. You can specifically invoke an exception to retrieve an Error.getStackTrace() to check who's calling. You can check the environment for certain class definitions and build in a couple of dummy classes just for that. You can check the stage if the display list hierarchy is correct. And of course don't keep any string literals you use there in a plain form. And of course protect all of above heavily.