Search code examples
javascriptjsonhttpbigdatadata-transfer

Transferring large amounts of json over http


I have large amounts (gigabytes) of json data that I would like to make available via a restful web service. The consumer of the data will be another service, and it will all be happening on a server (so the browser is not involved). Is there a practical limit on how much data can be transferred over http? Will http timeouts start to occur, or is that more a function of a browser?


Solution

  • There's no size limit for HTTP body. Just like downloading a huge file through web browser. And timeout is a setting of socket connection, over which HTTP is built, so it is not a browser specified feature.

    However, I've met the same issue with transporting quite large json object. What need to be considered are network load, serialize/deserialize time, and memory cost. The whole process is slow (2GB of data, via intranet, using JSON.NET and some calculation we take 2-3 minutes) and it costs quite large memory. Fortunately, we just need to do that once everyday and it is a back end process. So we don't pay more attention on it. We just use sync mode for HTTP connection and set a long timeout value to prevent timeout exception (Maybe async is a good choice).

    So I think it depends on your hardware and infrastructure.