Cannot convert big image with Golangs ImageMagick library

Question

I'm trying to load a big image (780MB) with Golangs ImageMagick library and change the format of the image from PNG to PTIF (pyramidal tiff). When I run the program different things can happen:

• it will sometimes close Vscode (high memory consuption)
• just hang. When it hangs it doesn't stop the program but it also doesn't use any resources anymore (watching htop).
• Sometimes I get the error runtime error: gobytes: length out of range this seems to happen when it gets bigger than the value in maxAlloc in cgo (ImageMagick is using C code). The error comes from https://go.dev/src/runtime/string.go line 304.
• It will just stop debugging after a while with the message in Vscode "Process x has exited with status -9"

When I set breakpoints it goes well until byteBuf := mim.mw.GetImageBlob() in the StoreImage function.

I have 16GB RAM on a Lenovo Linux Laptop and I already increased my swapfile to 8GB, but this only helped me with changing the format in the CLI with ImageMagick (convert collage.png test.ptif), it still doesn't work in the app. I'm guessing it is a memory issue but shouldn't 16GB be enough for a 780MB image? Any suggestions? The code works with smaller images.

package main

import (
    "bytes"
    "fmt"
    "io"
    "log"
    "os"

    "gopkg.in/gographics/imagick.v3/imagick"
)

type MyImageMagick struct {
    mw *imagick.MagickWand
}

func (mim *MyImageMagick) convert() {
    f, err := os.Open("collage.png")
    if err != nil {
        log.Fatal(err)
    }
    if err := mim.LoadImage(f); err != nil {
        log.Fatal(err)
    }
    r, err := mim.StoreImage("ptif")
    if err != nil {
        log.Fatal(err)
    }
    t, err := os.Create("testptif")
    if err != nil {
        log.Fatal(err)
    }
    n, err := io.Copy(t, r)
    if err != nil {
        log.Fatal(err)
    }
}
func (mim *MyImageMagick) LoadImage(reader io.Reader) error {
    var buf bytes.Buffer
    if _, err := buf.ReadFrom(reader); err != nil {
        return err
    }
    if err := mim.mw.ReadImageBlob(buf.Bytes()); err != nil {
        return err
    }
    return nil
}
func (mim *MyImageMagick) StoreImage(format string) (io.Reader, error) {
    if err := mim.mw.SetImageFormat(format); err != nil {
        return nil, err
    }
    byteBuf := mim.mw.GetImageBlob()
    return bytes.NewReader(byteBuf), nil
}
func main() {
    imagick.Initialize()
    defer imagick.Terminate()
    mim := &MyImageMagick{mw: imagick.NewMagickWand()}
    mim.convert()
}

Answer 1

libvips can do this kind of processing quickly and using relatively little memory. For example:

$ ls -l x.png
-rw-r--r-- 1 john john 995241537 Mar  8 09:01 x.png
$ vipsheader x.png
x.png: 15000x26319 uchar, 3 bands, srgb, pngload
$ /usr/bin/time -f %M:%e vips copy x.png x.tif[pyramid,tile,compression=jpeg]
220776:10.68

So a 1gb PNG file (15,000 x 26,000 pixels) converted to a pyramidal TIFF in under 11 seconds and needed 220mb of memory.

There are several Go bindings, but all of them seem to have various issues, unfortunately. Perhaps you could just shell out to the vips command? Conversion to pyramidal tiff is eg.:

vips copy input-file output-file.tiff[pyramid,tile,compression=jpeg]

You can set various other options for the tiff write:

https://www.libvips.org/API/current/VipsForeignSave.html#vips-tiffsave