Handling large http response using boost::beast

Question

The following code use to get http response message:

  boost::beast::tcp_stream stream_;

  boost::beast::flat_buffer buffer;
  boost::beast::http::response<boost::beast::http::dynamic_body> res;
  boost::beast::http::read(stream_, buffer, res);

However, In some cases, based on the preceding request, I can expect that the response message body will include large binary file.

Therefore, I’d like to read it directly to the filesystem and not through buffer variable to avoid excessive use of process memory. How can it be done ?

in Objective-c framework NSUrlSession there's an easy way to do it using NSURLSessionDownloadTask instead of NSURLSessionDataTask, so I wonder if it's also exist in boost.

Thanks !

Answer 1

In general, you can use the http::buffer_body to handle arbitrarily large request/response messages.

If you specifically want to read/write from a filesystem file, you can have the http::file_body instead.

Full Demo buffer_body

The documentation sample for buffer_body is here https://www.boost.org/doc/libs/1_77_0/libs/beast/doc/html/beast/using_http/parser_stream_operations/incremental_read.html.

Using it to write to std::cout: Live On Coliru

#include <boost/asio.hpp>
#include <boost/asio/ip/tcp.hpp>
#include <boost/beast.hpp>
#include <boost/beast/websocket.hpp>
#include <iostream>

namespace net       = boost::asio;
namespace beast     = boost::beast;
namespace http      = beast::http;
using tcp           = net::ip::tcp;
using socket_t      = tcp::socket;

/*  This function reads a message using a fixed size buffer to hold
    portions of the body, and prints the body contents to a `std::ostream`.
*/

template<
    bool isRequest,
    class SyncReadStream,
    class DynamicBuffer>
void
read_and_print_body(
    std::ostream& os,
    SyncReadStream& stream,
    DynamicBuffer& buffer,
    beast::error_code& ec)
{
    http::parser<isRequest, http::buffer_body> p;
    http::read_header(stream, buffer, p, ec);
    if(ec)
        return;
    while(! p.is_done())
    {
        char buf[512];
        p.get().body().data = buf;
        p.get().body().size = sizeof(buf);
        http::read(stream, buffer, p, ec);
        if(ec == http::error::need_buffer)
            ec = {};
        if(ec)
            return;
        os.write(buf, sizeof(buf) - p.get().body().size);
    }
}

int main() {
    std::string host = "173.203.57.63"; // COLIRU 20210901
    auto const port  = "80";

    net::io_context ioc;
    tcp::resolver   resolver{ioc};

    socket_t s{ioc};
    net::connect(s, resolver.resolve(host, port));

    write(s, http::request<http::empty_body>{http::verb::get, "/", 11});

    beast::error_code  ec;
    beast::flat_buffer buf;

    read_and_print_body<false>(std::cout, s, buf, ec);
}

Full file_body example

This is much shorter, writing to body.html:

Live On Coliru

#include <boost/asio.hpp>
#include <boost/asio/ip/tcp.hpp>
#include <boost/beast.hpp>
#include <boost/beast/websocket.hpp>
#include <iostream>

namespace net       = boost::asio;
namespace beast     = boost::beast;
namespace http      = beast::http;
using tcp           = net::ip::tcp;
using socket_t      = tcp::socket;

int main() {
    std::string host = "173.203.57.63"; // COLIRU 20210901
    auto const port  = "80";

    net::io_context ioc;
    tcp::resolver   resolver{ioc};

    socket_t s{ioc};
    net::connect(s, resolver.resolve(host, port));

    write(s, http::request<http::empty_body>{http::verb::get, "/", 11});

    beast::error_code  ec;
    beast::flat_buffer buf;
    http::response<http::file_body> res;
    res.body().open("body.html", beast::file_mode::write_new, ec);
    if (!ec.failed())
    {
        read(s, buf, res, ec);
    }

    std::cout << "Wrote 'body.html' (" << ec.message() << ")\n";
    std::cout << "Headers " << res.base() << "\n";
}

Prints

Wrote 'body.html' (Success)
Headers HTTP/1.1 200 OK 
Content-Type: text/html;charset=utf-8
Content-Length: 8616
Server: WEBrick/1.4.2 (Ruby/2.5.1/2018-03-29) OpenSSL/1.0.2g
Date: Wed, 01 Sep 2021 19:52:20 GMT
Connection: Keep-Alive

With file body.html; wc body.html showing:

body.html: HTML document, ASCII text, with very long lines
 185  644 8616 body.html

Beyond: streaming to child processes and streaming processing

I have an advanced example of that here: How to read data from Internet using muli-threading with connecting only once?.