Search code examples
machine-learningkerasdeep-learningneural-networkdata-augmentation

Why do we need test_generator and val_generator for data augmentation


Data augmentation is applied for training only. I'm wondering why several tutorials create test_generator and val_generator. Why don't we create only train_generator.


Solution

  • Actually, it is a pretty good practice that you separate train data and validation data. If you just create 1 generator, there is a pretty high chance that you validate your model with the same augmented data, which introduce a bias to your accuracy. Moreover, normally we use data augmentation when we have small number training data which makes thing even worst and end up with highly bias model. Therefore, we should separate the data and make sure the your model have not been exposed to any type of validation data so that it does not add any bias to your performance.

    For example, you may end up training model with picture-1 rotated clockwise and validate model with picture-1 rotated anti-clockwise. So, your validation accuracy that we normally use for determine overfitting is biased and you may end up with overfitted model without knowing when does it happens during training.