What's the cleanest and most efficient way to pass two stereo images to a loss function in Keras?

Question

First off, why am I using Keras? I'm trying to stay as high level as possible, which doesn't mean I'm scared of low-level Tensorflow; I just want to see how far I can go while keeping my code as simple and readable as possible.

I need my Keras model (custom-built using the Keras functional API) to read the left image from a stereo pair and minimize a loss function that needs to access both the right and left images. I want to store the data in a tf.data.Dataset.

What I tried:

1. Reading the dataset as (left image, right image), i.e. as tensors with shape ((W, H, 3), (W, H, 3)), then use function closure: define a keras_loss(left_images) that returns a loss(y_true, y_pred), with y_true being a tf.Tensor that holds the right image. The problem with this approach is that left_images is a tf.data.Dataset and Tensorflow complains (rightly so) that I'm trying to operate on a dataset instead of a tensor.

2. Reading the dataset as (left image, (left image, right image)), which should make y_true a tf.Tensor with shape ((W, H, 3), (W, H, 3)) that holds both the right and left images. The problem with this approach is that it...does not work and raises the following error:

   ValueError: Error when checking model target: the list of Numpy arrays 
   that you are passing to your model is not the size the model expected. 
   Expected to see 1 array(s), for inputs ['tf_op_layer_resize/ResizeBilinear'] 
   but instead got the following list of 2 arrays: [<tf.Tensor 'args_1:0' 
   shape=(None, 512, 256, 3) dtype=float32>, <tf.Tensor 'args_2:0' 
   shape=(None, 512, 256, 3) dtype=float32>]...
   

So, is there anything I did not consider? I read the documentation and found nothing about what gets considered as y_pred and what as y_true, nor about how to convert a dataset into a tensor smartly and without loading it all in memory.

My model is designed as such:

 def my_model(input_shape):
     width = input_shape[0]
     height = input_shape[1]
     inputs = tf.keras.Input(shape=input_shape)
     # < a few more layers >
     outputs = tf.image.resize(tf.nn.sigmoid(tf.slice(disp6, [0, 0, 0, 0], [-1, -1, -1, 2])), tf.Variable([width, height]))
     model = tf.keras.Model(inputs=inputs, outputs=outputs)
     return model

And my dataset is built as such (in case 2, while in case 1 only the function read_stereo_pair_from_line() changes):

def read_img_from_file(file_name):
    img = tf.io.read_file(file_name)
    # convert the compressed string to a 3D uint8 tensor
    img = tf.image.decode_png(img, channels=3)
    # Use `convert_image_dtype` to convert to floats in the [0,1] range.
    img = tf.image.convert_image_dtype(img, tf.float32)
    # resize the image to the desired size.
    return tf.image.resize(img, [args.input_width, args.input_height])


def read_stereo_pair_from_line(line):
    split_line = tf.strings.split(line, ' ')
    return read_img_from_file(split_line[0]), (read_img_from_file(split_line[0]), read_img_from_file(split_line[1]))

# Dataset loading
list_ds = tf.data.TextLineDataset('test/files.txt')
images_ds = list_ds.map(lambda x: read_stereo_pair_from_line(x))
images_ds = images_ds.batch(1)

Answer 1

Solved. I just needed to read the dataset as (left image, [left image, right image]) instead of (left image, (left image, right image)) i.e. make the second item a list and not a tuple. I can then access the images as input_r = y_true[:, 1, :, :] and input_l = y_true[:, 0, :, :]