How can I train a deep learning model with coordinates X, Y, and images?

Question

My task is cephalometric landmark localization. I have images path with coordinates show in this data frame.

filename	X1	Y1
/Images_data/binary0006.png	89	80
/Images_data/binary0008.png	37	70
/Images_data/binary0007.png	50	76
/Images_data/binary0003.png	55	92
/Images_data/binary0005.png	91	64
/Images_data/binary0004.png	100	76

How to prepare dataset for training in model.fit ? I try to create images dataset for training with ImageDataGenerator.

  train_ds = tf.keras.preprocessing.image_dataset_from_directory(
  data_dir,
  label_mode=None,
  validation_split=0.2,
  subset="training",
  seed=123,
  image_size=(img_height, img_width),
  batch_size=batch_size)

But now I'm stuck, because I don't know How to match coordinate with images.

Answer 1

Use the flow_from_dataframe method of ImageDataGenerator.

1. Load your DataFrame containing the file paths and coordinates.

import pandas as pd
data_dir = "images/"
df = pd.read_csv("dataframe.csv")
df = df.sample(frac=1, random_state=9)
print(df)

2. Create ImageDataGenerator

from tensorflow.keras.preprocessing.image import ImageDataGenerator
datagen = ImageDataGenerator(
    rescale=1./255,
)

3. Flow from DataFrame

from tensorflow.keras.preprocessing.image import ImageDataGenerator
img_height, img_width = 100, 100
batch_size = 32
train_generator = datagen.flow_from_dataframe(
    dataframe=df,
    directory=data_dir,
    x_col="x_col",
    y_col=["X1", "Y1"],
    target_size=(img_height, img_width),
    batch_size=batch_size,
    class_mode="raw",
    shuffle=True,
    seed=9,
    subset="training"
)

Here, x_col is set to "x_col" to specify the column containing file paths, and y_col is set to ["X1", "Y1"] to specify the columns containing the coordinates. The class_mode is set to "raw" to indicate that the labels are continuous values.

Now, you can use train_generator in the model.fit function:

model.fit(train_generator, epochs=num_epochs, ...)