How to decompress XZ data from a hyper::Response on the fly?

Question

I'm downloading an XZ file with hyper, and I would like to save it to disk in decompressed form by extracting as much as possible from each incoming Chunk and writing results to disk immediately, as opposed to first downloading the entire file and then decompressing.

There is the xz2 crate that implements the XZ format. However, its XzDecoder does not seem to support a Python-like decompressobj model, where a caller repeatedly feeds partial input and gets partial output.

Instead, XzDecoder receives input bytes via a Read parameter, and I'm not sure how to glue these two things together. Is there a way to feed a Response to XzDecoder?

The only clue I found so far is this issue, which contains a reference to a private ReadableChunks type, which I could in theory replicate in my code - but maybe there is an easier way?

Answer 1

XzDecoder does not seem to support a Python-like decompressobj model, where a caller repeatedly feeds partial input and gets partial output

there's xz2::stream::Stream which does exactly what you want. Very rough untested code, needs proper error handling, etc, but I hope you'll get the idea:

fn process(body: hyper::body::Body) {
    let mut decoder = xz2::stream::Stream::new_stream_decoder(1000, 0).unwrap();
    body.for_each(|chunk| {
        let mut buf: Vec<u8> = Vec::new();
        if let Ok(_) = decoder.process_vec(&chunk, &mut buf, Action::Run) {
            // write buf to disk
        }
        Ok(())
    }).wait().unwrap();
}