Search code examples
zipcompressiontoit

Compressing data in Toit


The Toit library contains the zip.toit module, which allows to hope that the system supports data compression and decompression. Unfortunately, there are no examples. Is it possible to give the simplest example of compression and decompression of data, such as strings or binary arrays?


Solution

  • The zlib library currently only supports compression.

    Fundamentally you have to:

    1. create a compressor (zlib, or gzip).
    2. feed the data in one task
    3. read/use the compressed data in another task.

    Here is an example:

    import zlib
    import bytes
    import monitor
    
    main:
      // We use a semaphore to let the main task know when all the
      // compressed data has been handled.
      // If the data is just sent over the network, then the semaphore
      // wouldn't be necessary. The second task (`t`) would just finish
      // once all data has been processed.
      done := monitor.Semaphore
    
      // We use a byte-buffer to accumulate all the data. If the data is
      // to be sent somewhere, then it's better to just do that directly,
      // instead of using memory here.
      accumulator := bytes.Buffer
    
      // There are other encoders as well, but the gzip-encoder has
      // the advantage that it produces `.gz` compatible data.
      compressor := zlib.RunLengthGzipEncoder
    
      // We create a second task that takes out the already compressed
      // data of the compressor.
      t := task::
        while data := compressor.read:
          accumulator.write data
        // We increment the semaphore, so that the other (original) task
        // knows that we are done processing the data.
        done.up
    
      // In this task we now add data. As example, it's just "foo", but any
      // string or byte-array would work.
      // Can be called multiple times.
      compressor.write "foo"
    
      // The compressor must be closed.
      // This flushes the last remaining data, and lets the reading
      // task know that it is done.
      compressor.close
    
      // We wait for the reading task to signal that all data is handled.
      done.down
    
      // Show the data that was accumulated.
      // You would normally just send it somewhere, and not print it.
      print accumulator.buffer.size
      print accumulator.buffer