Search code examples
cntkresnet

ResNet train and evaluation result don't match on CNTK

I have an image classify application with 6 classes and image size 128x64x3. I used the ResNet 20 model to train with error rate about 2%. However, the evaluation result is over 20% on the same set of train images. The evaluation used the nuget CNTK.CPUOnly/2.0 with onEvaluateAnColorImage function. The questions are:

  1. Does ResNet work on CPUOnly mode for evaluation? The model was train with GPU on.
  2. Does ResNet require mean file? I see some models with mean file and some don't.
  3. Any particular reason that could cause the difference train and evaluation results?

Thanks in advance, Terry

Solution

  • Resnet trained with GPU should work with CPUOnly evaluation. If possible, you can use CNTK.GPU Nuget package and run your application on a GPU device to check whether you still see the difference in accuracy. If yes, please let us know and we will investigate it.

    If you are using mean file during training, then you should substract the mean value during evaluation too.

    Another reason could cause difference could be image preprocessing. The training usaully uses readers which will automatically perform resization and channel transformation, but you have to do this by yourself for evaluation, since no reader is involved. You can find image evaluation examples here.

    Thanks,