The problem follows. It is necessary to work with very large binary images (100000x100000 pixels). Initially did it using Qt's QImage class, it supports Format_Mono format that stores an image as a 1-bit per pixel. And in general, everything was fine, until it turned out that QPainter has limited rasterizer and draw on images whose size is more short (32767x32767) can not be, it just cut off.
I was not able to combine images by more than 32767x32767. Then, I began to look closely to individual libraries. The OpenCV, as I understand it, does not support this format. Regarding ImageMagick, it supports the construction of the image as one-bit per pixel and save it in the same format. However, while working with the image is still stored as an 8bit per pixel and hence there arises a shortage of RAM. Then I decided to try CImg, but it don't suppor 1bbp format, as i understand:
the overall size of the used memory for one instance image (in bytes) is then 'width x height x depth x dim x sizeof (T)
Where sizeof (T) of course can not be less than sizeof (char)...
It was interesting how QImage in principle works with its Format_Mono format, but honestly, I was tangled in the source code.
So, i have the next question. Is there a library that implemented the ability to create and work with binary images, and in this case they really are stored as a 1-bit per pixel in RAM?
libvips can process huge 1 bit images. It unpacks them to one pixel per byte for processing, but it only keeps the part of the image currently being processed in memory, so you should be OK.
For example, this tiny program makes a 100,000 x 100,000 pixel black image, then pastes in all of the images from the command-line at random positions:
#!/usr/bin/env python
import sys
import random
import gi
gi.require_version('Vips', '8.0')
from gi.repository import Vips
# this makes a 8-bit, mono image of 100,000 x 100,000 pixels, each pixel zero
im = Vips.Image.black(100000, 100000)
for filename in sys.argv[2:]:
tile = Vips.Image.new_from_file(filename, access = Vips.Access.SEQUENTIAL)
im = im.insert(tile,
random.randint(0, im.width - tile.width),
random.randint(0, im.height - tile.height))
im.write_to_file(sys.argv[1])
I can run the program like this:
$ vipsheader wtc.tif
wtc.tif: 9372x9372 uchar, 1 band, b-w, tiffload
$ mkdir test
$ for i in {1..1000}; do cp wtc.tif test/$i.tif; done
$ time ./insert.py x.tif[bigtiff,squash,compression=ccittfax4] test/*.tif
real 1m31.034s
user 3m24.320s
sys 0m7.440s
peak mem: 6.7gb
The [] on the output filename set image write options. Here I'm enabling fax compression and setting the squash option. This means 8-bit one band images should be squashed down to 1 bit for write.
The peak mem result is from watching RES in top. 6.7gb is rather large unfortunately, as it's having to keep input buffers for each of the 1,000 input images.
If you use tiled 1-bit tiff, you can remove the access = option and use operators that need random access, like rotate. If you try to rotate a strip tiff, vips will have to decompress the whole image to a temporary disk file, which you probably don't want.
vips has a reasonable range of standard image processing operators, so you may be able to do what you want just glueing them together. You can add new operators in C or C++ if you want.
That example is in Python for brevity, but you can use Ruby, PHP, C, C++, Go, JavaScript or the command-line if you prefer. It comes with all Linuxes and BSDs, it's on homebrew, MacPorts and fink, and there's a Windows binary.