Search code examples
image-processingvips

libvips rotate is throwing no space left on device


I am using libvips to rotate the images. I am using a VM that have 3002 MB Ram and 512MB temp storage.The AWS Lambda Machine.

The command I running to rotate images is

vips rot original.jpg rotated.jpg d90

It throwing the following error

Exit Code: 1, Error Output: ERROR: wbuffer_write: write failed unix error: No space left on device

The jpg image is arround 10Mb.


Solution

  • Here's how libvips will rotate your jpg image.

    90 degree rotate requires random access to the image pixels, but JPEG images can only be read strictly top-to-bottom, so as a first step, libvips has to unpack the JPG to a random access format. It uses vips (.v) format for this, which is pretty much a C array with a small header.

    For images under 100mb (you can change this value, see below) decompressed, it will unpack to a memory buffer. For images over 100mb decompressed, it will unpack to a temporary file in /tmp (you can change this, see below).

    Next, it does the rotate to the output image. It can do this as a single streaming operation, so it will typically need enough memory for 256 scanlines on the input image, and 256 on the output, so around another 30mb or so in this case, plus some more working area for each thread.

    In your specific case, the input image is being decompressed to a temporary file of 30,000 x 10,000 x 3 bytes, or about 900mb. This is way over the 512mb you have in /tmp, so the operation fails.

    The simplest solution is to force the loader to load via a memory buffer. If I try:

    $ vipsheader x.jpg 
    x.jpg: 30000x10000 uchar, 3 bands, srgb, jpegload
    $ time vips rot x.jpg y.jpg d90 --vips-progress --vips-leak
    vips temp-3: 10000 x 30000 pixels, 8 threads, 128 x 128 tiles, 256 lines in buffer
    vips x.jpg: 30000 x 10000 pixels, 8 threads, 30000 x 16 tiles, 256 lines in buffer
    vips x.jpg: done in 0.972s          
    vips temp-3: done in 4.52s          
    memory: high-water mark 150.43 MB
    real    0m4.647s
    user    0m5.078s
    sys 0m8.418s
    

    The leak and progress flags make vips report some stats. You can see the initial decompress to the temporary file is talking 0.97s, the rotate to the output is 4.5s, it needs 150mb of pixel buffers and 900mb of disc.

    If I raise the threshold, I see:

    $ time VIPS_DISC_THRESHOLD=1gb vips rot x.jpg y.jpg d90 --vips-progress --vips-leak
    vips temp-3: 10000 x 30000 pixels, 8 threads, 128 x 128 tiles, 256 lines in buffer
    vips x.jpg: 30000 x 10000 pixels, 8 threads, 30000 x 16 tiles, 256 lines in buffer
    vips x.jpg: done in 0.87s          
    vips temp-3: done in 1.98s          
    memory: high-water mark 964.79 MB
    real    0m2.039s
    user    0m3.842s
    sys 0m0.443s
    

    Now the second rotate phase is only 2s since it's just reading memory, but memory use has gone up to around 1gb.

    This system is introduced in the libvips docs here:

    http://jcupitt.github.io/libvips/API/current/How-it-opens-files.md.html