Search code examples
ioscompressionlz4

Decoding the LZ4 output from iOS's compression_encode_buffer


I've got an iOS app compressing a bunch of small chunks of data. I use compression_encode_buffer running in LZ4 mode to do it so that it is fast enough for my needs.

Later, I take the file[s] I made and decode them on a non-Apple device. Previously I'd been using their ZLIB compression mode and could successfully decode it in C# with System.IO.Compression.DeflateStream.

However, I'm having a hell of a time with the LZ4 output. Based on the LZ4 docs here, Apple breaks the stream into a bunch of blocks, each starting with a 4-byte magic number, 4-byte decompressed size, and 4-byte compressed size. All that makes sense, and I'm able to parse the file into its consituent raw-LZ4 chunks. Each chunk in the buffer iOS outputs decompresses to about 65,635 bytes, and there's about 10 of them in my case.

But then: I have no idea what to DO with the LZ4 chunks I'm left with. I've tried decoding them with LZ4net's LZ4.LZ4Stream, LZ4net's LZ4.LZ4Codec (it manages the first block, but then fails when I feed in the 2nd one). I've also tried several C++ libraries to decode the data. Each of them seem to be looking for a header that the iOS compression functions have encoded in a non-standard way.


Solution

  • Answering my own: Apple's LZ4 decompressor (with necessary modifications to handle their raw storage format) is here: https://opensource.apple.com/source/xnu/xnu-3789.21.4/osfmk/vm/lz4.c.auto.html

    Edit afterwards: I actually wasn't able to get this working, but I didn't spend much time on it because I found Apple's LZFSE decompressor.

    LZFSE Decompressor can be found here: https://github.com/lzfse/lzfse