Generate image file without keeping the whole image in memory

Question

I wrote a program to generate RGB color values using three equations and write them to a data file:

from struct import pack

#User-set variables
size = [16384, 16384]
center = [0, 0]
r_equation = "x ^ y"
g_equation = "x - y"
b_equation = "x + y"
data_path = "data.dat"

#Calculate x and y offset
offset = [center[0] - (size[0] // 2), center[1] - (size[1] // 2)]

#Compile code for calculating RGB values
r_code = compile(r_equation, "r_equation", "eval")
g_code = compile(g_equation, "g_equation", "eval")
b_code = compile(b_equation, "b_equation", "eval")

data_file = open(data_path, "wb")

#Pack image size as first 4 bytes
data_file.write(pack("HH", *size))

#Iterate through the graph
for y in range(offset[1], size[1] + offset[1]):
    for x in range(offset[0], size[0] + offset[0]):
        #Calculate the RGB values
        rgb = [int(eval(r_code)) % 256,
               int(eval(g_code)) % 256,
               int(eval(b_code)) % 256]
        #Pack the RGB values as 3 bytes
        data_file.write(pack("BBB", *rgb))

data_file.close()

I want to create an image from the data file, but I don't have enough RAM to load the entire image. I wrote this program to read pieces of the file:

from struct import unpack

#User-set variables
data_path = "data.dat"
image_path = "image.xxx"

data_file = open(data_path, "rb")

#Unpack first 4 bytes to get image size
size = unpack("HH", data_file.read(4))

#create_image(size, image_path)

for data in data_file.read(1048576 * 3): #Read 1M pixels at a time
    #write_to_image(data)

If I try to use PIL to write the entire image file, it soon runs out of memory. Is there a way to write the image file in pieces so one piece is in memory at a time?

Answer 1

Here's your program (I think!) done with pyvips:

#!/usr/bin/python3

import sys
import pyvips

# make a huge two-band image where each pixel has the value of its (x, y)
# coordinate
xy = pyvips.Image.xyz(16384, 16384)

# subtract the half way point, so each pixel is now -8192 to +8191
xy -= [8192, 8192]

# compute three one-band images from (x, y) ... image[n] gets the nth band
r = xy[0] ^ xy[1]
g = xy[0] + xy[1]
b = xy[0] - xy[1]

# join as an RGB image, modulo 256
rgb = r.bandjoin([g, b]).copy(interpretation="srgb") & 0xff

rgb.write_to_file(sys.argv[1])

On this modest laptop, I see:

$ /usr/bin/time -f %M:%e ./pattern.py x.jpg
136712:5.81s

6s and 137mb of memory.