How can I prevent OOMs when resizing JPGs with ImageMagick (without increasing memory)?

Question

It seems the -limit option is ignored when resizing JPGs with ImageMagick.

The following command uses docker to simulate running ImageMagick on a machine with limited free memory (100MB): the ImageMagick command is passed a 4958 × 6198 JPEG (original file size: 5.5MB) to resize, with -limit memory 5MiB.

The result is that ImageMagick gets killed by the container's OOM Killer, implying ImageMagick attempts to allocate over ~100MB instead of staying within the ~5MB limit it was instructed to. If you increase (or remove) the --memory 100m flag on docker run, the command will succeed:

# Download a 5.5MB JPEG (4958 × 6198)
curl 'https://images.unsplash.com/photo-1534970028765-38ce47ef7d8d?ixlib=rb-1.2.1&q=80&fm=jpg&crop=entropy&cs=tinysrgb&dl=trail-5yOnGsKUNGw-unsplash.jpg' \
     -o 5500kb-image.jpg

docker run --memory 100m \
           -v $(pwd):/imgs \
           dpokidov/imagemagick \
           /imgs/5500kb-image.jpg \
           -debug All -limit memory 5MiB -limit map 10MiB -resize 400 \
           /imgs/5500kb-image-resized.jpg

How can I memory-bound ImageMagick when resizing JPEGs?

Answer 1

I think you have the arguments in the wrong order.

If you're on linux, /usr/bin/time is very convenient for measuring peak memory use. You are running this command:

$ /usr/bin/time -f %M:%e \
    convert \
        5500kb-image.jpg \
        -limit memory 5MiB -limit map 10MiB \
        -resize 400 5500kb-image-resized.jpg
346060:2.37

ie. 350mb of memory and 2.37s of CPU time.

If you put the limit first, you see:

$ /usr/bin/time -f %M:%e \
    convert \
        -limit memory 5MiB -limit map 10MiB \
        5500kb-image.jpg -resize 400 5500kb-image-resized.jpg
105468:2.64

Now it's just 105mb peak, and about the same runtime. You need to set the limits before you load the image.

As Mark says, vipsthumbnail is quite a bit faster and less memory hungry. I see:

$ /usr/bin/time -f %M:%e \
    vipsthumbnail 5500kb-image.jpg --size 400 -o 5500kb-image-resized.jpg
133676:0.28

130mb and 0.28s.

That's still rather high. Your image is a progressive JPG --- these must be loaded entirely into memory before thumbnailing can start, and it's not possible to exploit things like jpeg shrink-on-load.

If I convert your image to a regular jpeg, I see:

$ /usr/bin/time -f %M:%e \
    vipsthumbnail 5500kb-image-regular.jpg --size 400 -o 5500kb-image-resized.jpg
40200:0.09

40mb and 0.1s of CPU.

vipsthumbnail can resize very large images with only a small amount of memory, for example:

$ vipsheader st-francis.jpg 
st-francis.jpg: 30000x26319 uchar, 3 bands, srgb, jpegload
$ /usr/bin/time -f %M:%e \
    vipsthumbnail st-francis.jpg --size 400 -o 5500kb-image-resized.jpg
49692:2.52

50mb and 2.5s.