Keras data augmentation pipeline for image segmentation dataset (image and mask with same manipulation)
I am building a preprocessing and data augmentation pipeline for my image segmentation dataset There is a powerful API from keras to do this but I ran into the problem of reproducing same augmentation on image as well as segmentation mask (2nd image). Both images must undergo the exact same manipulations. Is this not supported yet?
https://www.tensorflow.org/tutorials/images/data_augmentation
Example / Pseudocode
data_augmentation = tf.keras.Sequential([
layers.experimental.preprocessing.RandomFlip(mode="horizontal_and_vertical", seed=SEED_VAL),
layers.experimental.preprocessing.RandomRotation(factor=0.4, fill_mode="constant", fill_value=0, seed=SEED_VAL),
layers.experimental.preprocessing.RandomZoom(height_factor=(-0.0,-0.2), fill_mode='constant', fill_value=0, seed=SEED_VAL)])
(train_ds, test_ds), info = tfds.load('somedataset', split=['train[:80%]', 'train[80%:]'], with_info=True)
This code does not work but illustrates how my dream api would work:
train_ds = train_ds.map(lambda datapoint: data_augmentation((datapoint['image'], datapoint['segmentation_mask']), training=True))
Alternative
The alternative is to code a custom load and manipulation / randomization method as is proposed in the image segmentation tutorial (https://www.tensorflow.org/tutorials/images/segmentation)
Any tips on state of the art data augmentation for this type of dataset is much appreciated :)
You can try with external libraries for extra image augmentations. These links may help for image augmentation along with segmentation mask,
albumentations
https://github.com/albumentations-team/albumentations
https://albumentations.ai/docs/getting_started/mask_augmentation/
imgaug
https://github.com/aleju/imgaug
