I have a pose estimation model pretrained on a dataset in which hands are in its nartural color. I want to finetune that model on the dataset of hands of surgeons doing surgeries. Those hands are in surgical gloves so the image of the hands are a bit different than normal hands.
Does this difference in hand colors affect the model performance? If I can make images of those surgical hands more like normal hands, will I get better performance?
Well, it depends on what your pre-trained model has learned to capture from the pre-training (initial) dataset. Suppose your model had many feature maps and not enough skin color variation in your pre-training dataset (leads to overfitting issues). In that case, your model has likely "taken the path of least resistance" and exploited that to learn feature maps that rely on the color space as means of feature extraction (which might not generalize well due to color differences).
The more your pre-training dataset match/overlap with your target dataset, the better the effects of transfer learning will be. So yes, there is a very high chance that making your target dataset (surgical hands) look more similar to your pre-training dataset will positively impact your model's performance. Moreover, I would conjecture that introducing some color variation (e.g., Color Jitter augmentation) in your pre-training dataset could also help your model generalize to your target dataset.
