Search code examples
c++tcpnetwork-programming

Sending large files over the network

I have few GB of files that I need to send over the network. I have to compress them and then send then through. I'm currently loading the whole file into memory, compress it and send it over. This works fine on my computer but it doesn't work good on customers infrastructure as the VM's they are running this on has limited amount of memory.

Right now what I do is as follow,

  1. Read the whole file, the files are in binary and I do fopen(), fread() until I read everything.
  2. Compress the loaded file in memory
  3. I get 256 Bytes at a time and send it over the network(The 256 Bytes is the max limit)

The other way would be to read 256 Bytes from the file, compress them and then send it over. But 256 raw bytes after the compression will become around around 200 bytes so on each send operation I miss 56 Bytes in the payload...

The project is written in C++ and for compression I'm using TinyLZMA library.

Is anyone has better way of doing this? Something that doesn't require loading the whole file into memory, or losing several bytes in each send operation?

Thanks in advance

Solution

  • I would give boost streams a try. It has builtin support for streaming lzma encoded content.

    The general flow would be as follows:

    • set up a ifstream with your file as input
    • create a filtering output streambuf
    • add a lzma compressor to the filter
    • add a sink to store the resulting compressed stream

    Use this example as a rough reference on how to do it. The sink could be adapted to the tutorial on writing a container sink. Your implementation would be called with a bunch of bytes and needs to store these bytes in a container. Once you have 256 bytes full you'd need to send it to the network. After the chunk is being send you give back control to the stream and receive more bytes.

    This way you would not need to read the whole file nor need to store the result in memory.