Stream in images to keras from disk to avoid blowing up the memory

Question

I have the following model that is supposed to zoom out (outpaint) images. I have a dataset of 20000 300x400 images. The problem is that neither my GPU nor RAM can hold all images at once. So this code simply overflows it and crashes my computer. So I tried using keras flow_from_directory which seemed to sort of do what I want. The problem is that it only works with class based labels. In fact it requires that the data be separated into sub folders, representing classes. This of course gives the input the wrong shape. It also has no way to set the zoomed out images as a label (since you can only pass a single generator to train on to the model). So how can I stream both the label images and input images from disk?

import os
import numpy as np
from keras.models import Sequential
from keras.layers import Conv2D, UpSampling2D
from PIL import Image
from numpy import asarray
    
# Define image size and number of channels
img_width = 400
img_height = 300
channels = 3

# Define the model architecture
model = Sequential()
model.add(Conv2D(32, (3, 3), activation='relu', padding='same', input_shape=(img_height, img_width, channels)))
model.add(Conv2D(64, (3, 3), activation='relu', padding='same'))
model.add(Conv2D(128, (3, 3), activation='relu', padding='same'))
model.add(Conv2D(64, (3, 3), activation='relu', padding='same'))
model.add(Conv2D(channels, (3, 3), activation='sigmoid', padding='same'))

# Compile the model
model.compile(optimizer='adam', loss='mse')

# Load the input and label images
input_dir = "input/"
label_dir = "label/"
input_files = os.listdir(input_dir)
label_files = os.listdir(label_dir)

# Create numpy arrays for the input and label data
input_data = np.zeros(((50), img_height, img_width, channels), dtype=np.float32)
label_data = np.zeros(((50), img_height, img_width, channels), dtype=np.float32)

# Load the input and label data into the numpy arrays
for i in range(50):
    # Import the necessary libraries

    # load the image and convert into
    # numpy array
    img = Image.open(input_dir+input_files[i])
    img2 = Image.open(label_dir+label_files[i])

    if(asarray(img).shape != (300,400,3)):
        continue
    # asarray() class is used to convert
    # PIL images into NumPy arrays
    input_data[i] = asarray(img)/255.0
    label_data[i]=asarray(img2)/255.0


model.fit(input_data, label_data, epochs=10, batch_size=32)

Answer 1

Heres how I ended up doing it. I loaded with two Data Generators side by side, making sure shuffle is deactivated. Then I simply comibined them. The images also need to be in subdirectories of their folder, for whatever reason:

# Augment the training data with random transformations
input_datagen = ImageDataGenerator(rescale=1./255)


# Use the flow_from_directory function to generate batches of data
input_datagen = input_datagen.flow_from_directory(input_dir,
                                                    target_size=(img_height, img_width),
                                                    batch_size=batch_size,
                                                    class_mode=None,
                                                    shuffle=False
                                                 )

# Augment the training data with random transformations
label_datagen = ImageDataGenerator(rescale=1./255)


# Use the flow_from_directory function to generate batches of data
label_datagen = label_datagen.flow_from_directory(label_dir,
                                                    target_size=(img_height, img_width),
                                                    batch_size=batch_size,
                                                  class_mode=None,
                                                  shuffle=False
                                                 )




# Train the model using the generators
model.fit_generator(zip(label_datagen,input_datagen),
                    steps_per_epoch=input_datagen.n//batch_size,
                    epochs=1)