Search code examples
pngsaveopenlayerslibpngprocessing-efficiency

write png quickly

Summary

I want to write a .png file as quickly as possible, without a concern for compression. That is, I don't care much about the file size, but I do care that the write happens as quickly as possible.

Motivation

I am making a web-based map application using OpenLayers on the client side and python/C++ on the back end. The application needs to be able to draw dynamic content quickly as the user moves around the map. I have both tile-based (256x256 tile) and single-image-based ("single tile") versions working, but in both cases the slowest part of the back-end render is actually saving the image as a png file (whether on-disk or in-memory). For instance, I may be able to generate a "raw", "tga", or "tiff" version of a certain view in about 200 ms, but it takes more like 1.2 seconds to generate the .png version just because the .png save takes almost a full second, whereas the time to actually save the other formats is 100 ms or less (and even though the "raw" file is five times the size of the .png file). And this file save time is also significantly more than the time to transfer the resulting image from the server to the client. (One important property of my app is that in general the "back end" will be running on the same machine as the browser, so transfer times are negligible, even for largeish files.)

I thought I could make .png writes fast (when using libpng from C++) by calling

    png_set_compression_level( png_ptr, 0 );

before calling any png_write_... functions. However, while that call does indeed seem to stop libpng from compressing the file (the resulting file is about the same size as the .raw file), it doesn't make saving the .png noticeably faster.

Please help

I need to use .png for these image, because I need them to be transparent overlays on top of the base map, and I need more than the 256 colors offered by GIF. OpenLayers is simply using html img tags, so my understanding is that I can only use valid img formats.

I would think there would be a way of writing a .png file quickly by not doing any real compression (I understand that .png is "always compressed" but I imagined this could include "null compression"). It seems like you should be able to write like a simple fixed header, followed by the uncompressed data, followed by some fixed footer. Or maybe that same idea but in a line-by-line way. The point being that I can do all sorts of looping through this 2.5 MB of raw data in memory in C++ very quickly, and can dump it to various file formats very quickly, so it seems like I should be able to dump it in a fixed, uncompressed .png format quickly also.

Does that sound right? Do you know where I can find examples of code that does that?

Solution

  • what you want is an implementation that is specialized for your purpose; you are going to have to write your own encoder. it's actually not too hard and the specs are free.

    the format isn't too complex and should be easy to implement an encoder

    note: all values are unsigned. Multiple-byte integers are in "network byte order" (most significant byte first).

    the format is composed of chunks. chunk structure:

    • length of chunk contents, 4 bytes
    • chunk identifier (ASCII), 4 bytes
    • chunk contents, "length of chunk contents" bytes
    • CRC of the identifier and the contents (i.e. excluding the length), 4 bytes

    your implementation should only need the magic number and three chunks:

    detailed layout:

    • { 137, 80, 78, 71, 13, 10, 26, 10 } (magic number), 8 bytes
    • byte length of IHDR chunk, 4 bytes (value: 13)
    • { 73, 72, 68, 82 } ("IHDR"), 4 bytes
    • width, 4 bytes
    • height, 4 bytes
    • bit depth (per color), 1 byte (8 = 24/32 bit color)
    • color type, 1 byte (2 = RGB)
    • compression method, 1 byte (0 = DEFLATE algo which allows no compression)
    • filter method, 1 byte (0 = no filter)
    • interlace method, 1 byte (0 = no interlace)
    • CRC of IHDR chunk, 4 bytes
    • byte length of IDAT chunk contents, 4 bytes
    • { 73, 68, 65, 84 } ("IDAT"), 4 bytes
    • raw image data with DEFLATE algo wrapping, "byte length of IDAT chunk contents" bytes
    • CRC of IDAT chunk, 4 bytes
    • byte length of IEND chunk, 4 bytes (value: 0)
    • { 73, 69, 78, 68 } ("IEND"), 4 bytes
    • CRC of IEND chunk, 4 bytes (can be precomputed)

    DEFLATE data encoding with no compression

    your data will be split into chunks of 65535 bytes the format is simple:

    • first X chunks
      • header, 1 byte (value: 0)
      • data, 65535 bytes
    • final chunk
      • header, 1 byte (value: 1)
      • data, 65535 bytes or less

    that's it

    so that is how you make a fast png file