Search code examples
javac#tcpstreamtcpclient

Read from TcpClient.GetStream() without knowing the length

I'm working on a tcp base communication protocol . As i know there are many ways to determine when to end reading.

  1. Closing the connection at the end of the message
  2. Putting the length of the message before the data itself
  3. Using a separator; some value which will never occur in the normal data (or would always be escaped somehow)

Typically i'm trying to send a file over the WiFi network (that may be Unstable and Low speed)

  • Cause of RSA and AES communication I don't like to close the connection each time (Can't use 1)
  • It's a large file that i cant predict the length of it so i cant act as method (Can't use 2)
  • Checking for something special when reading and escape it when writing need a lot of process (Can't use 3)
  • This method should be compatible with both c# and java.

What you suggest ?

More general problems :

How to identify end of InputStream in java

C# - TcpClient - Detecting end of stream?

More Iformation

I'm coding a TCP client server communication

At first server generates and sends a RSA public code to the client.

Then the client will generate AES(key,IV) and send it back using RSA encryption.

Till here everything is fine.

But i want to send a file over this network. here is my current packet EncryptUsingAES(new AES.IV(16 byte) +file.content(any size))

In the server i can't capture all the data sent by client. So i need to know how much data to read with (TcpClient.GetStream().read(buffer , 0 , buffersize) ) Current code:

List<byte> message = new List<byte>();
    int bytes = -1;
    do
    {
        byte[] buffer = new byte[bufferrSize];
        bytes = stream.Read(buffer, 0, bufferrSize);
        if (bytes > 0)
        {
            byte[] tmp = new byte[bytes];
            Array.Copy(buffer, tmp, bytes);
            message.AddRange(tmp);
        }
    } while (bytes == bufferrSize);
Solution

  • Your second method is the best one. Prefixing each packet with the packet's length will create a reliable message framing protocol which will, if done correctly, ensure that all your data is received even in the same size you sent it (that is, no partial data or data being lumped together).

    • Recommended packet structure:

      [Data length (4 bytes)][Header (1 byte)][Data (?? bytes)]

      - The header in question is a single byte you can use to indicate what kind of packet this is, so that the endpoint will know what to do with it.


    Sending files

    The sender of a file is in 90% of the cases aware of the amount of data it is about to send (after all, it usually has the file stored locally), which means there will be no problem knowing how much of the file has been sent or not.

    The method I use and recommend is that you start by sending an "info packet", which explains to the endpoint that it is about to receive a file and also how many bytes that file consists of. After that you start sending the actual data - most preferrably in chunks since it's inefficient to proccess the entire file at once (at least if it's a large file).

    • Always keep track of how many bytes of the file you've received so far. By doing so the receiver can automatically tell when it has received the whole file.
    • Send a file a few kilobytes at a time (I use 8192 bytes = 8 kB as a file buffer). That way you don't have to read the entire file into memory nor encrypt it all at the same time.


    Encrypting the data

    Dealing with encryption will not be a problem. If you use length-prefixing just encrypt the data itself and leave the data length header untouched. The data length header must then be generated by the size of the encrypted data, like so:

    1. Encrypt the data.
    2. Get the length of the encrypted data.

    3. Produce the following packet:

      [Encrypted data length][Encrypted data]

      (Insert a header byte in there if you need to)


    Receiving an encrypted file

    Receiving an encrypted file and knowing when everything has been received is infact not very hard. Assuming you're using the above the described method for sending the file, you would just have to:

    1. Receive the encrypted packet → decrypt it.
    2. Get the length of the decrypted data.
    3. Increment a variable keeping track of the amount of file-bytes received.
    4. If the received amount is equal to the expected amount: close the file.


    Additional resources/references

    You can refer to two of my previous answers that I wrote about TCP length-prefixed message framing: