How to start AI-Platform jobs automatically?

Question

I created a training job where I fetch my data from big query, perform training and deploy model. I would like to start training automatically in this two cases:

1. More than 1000 new rows added to the dataset
2. With a schedule (Ex, once a week)

I checked GCP Cloud Scheduler, but it seems its not suitable for my case.

Answer 1

Cloud Scheduler is the right tool to trigger your training on a schedule. I don't know what your blocker is!!

For your first point, you can't. You can't put a trigger (on BigQuery or on other database) to send an event after X new rows. For this, I recommend you to do this:

• Schedule a job with Cloud Scheduler (for example every 10 minutes)
• The job perform a request in BigQuery and check the number of line since the last training job (the date of the last training job must be somewhere, I recommend in another BigQuery table)
  • If the number of line is > 1000, trigger your running job
  • Else, exit the function

As you see, it's not so easy and there is several caveats:

• When you deploy your model, you also have to write the date of the latest training
• You have to perform several times the request into BigQuery. Partition correctly your table for limiting the cost

Does it make sense for you?

EDIT

gcloud command is a "simple" wrapper of API calls. Try to add the param --http-log to your gcloud command to see which API is called and with which params.

Anyway, you can start a job by calling this API, and if you want and example, use the --http-log param of gcloud SDK!