Search code examples
google-cloud-platformgoogle-cloud-automl

Pipeline Timeout Error when training using Google AutoML

Previously I successfully trained my small dataset using Google AutoML with decent accuracy. Now I want to train another dataset with 999.683 images (nearly the limit of images in 1 dataset) spread in 28 classes.

The training process took a very long time, after 24 hour of waiting Google sent me the error report

Due to an error, AutoML Vision was unable to process your dataset "My_Dataset". There will be no charge issued for this job.

And the error details on AutoML Vision only shows

Pipeline c350ef1471db4bdb800f743eef963aec timeout.

Anybody has experienced this? Is this the problem with my dataset? Or is it a problem on Google AutoML server side?

Solution

  • This is caused by using the free version (1 Node hour) on very large training dataset. The training process is not finished and when it exceeds 24 hour it will throw that error. By selecting the 24 Node hour option, it will continue the training process without throwing any error.

    Caution :

    • For my nearly 1 mil dataset, it took 2~3 days until I got email from Google indicating the training has finished.
    • It used all of my free 300$ credits