Search code examples
google-cloud-platformgoogle-bigquerygoogle-cloud-logginggoogle-cloud-monitoring

creating an alert when no data was uploaded to BigQuery table in GCP

I have a requirement to send an email notification whenever there is no data getting inserted into my BigQuery table. For this, I am using the Logging and Alerting mechanism But still, I am not able to receive any email. Here are the steps I followed:

  1. I had written a Query in Logs explorer as below:

enter image description here

  1. Now I had created a metric for those logs with Metric type COUNTER and in the filter section obviously I have given the above query.

  2. Now I created a policy in ALERTING under the MONITORING domain. And here is the screenshot attached. The alerting policy which I had selected is for the logging metrics which I had created before.

enter image description here

  1. And then a trigger as below:

enter image description here

And in the Notification channel, added my Email ID.

Can someone please help me if I am missing something? My requirement is to receive an alert when there is no data inserted into a Bigquery table for more than a day.

And also, I could see in Metrics Explorer, the metric which I created is not ACTIVE. Why so?

enter image description here

Solution

  • As mentioned in GCP docs:

    Metric absence conditions require at least one successful measurement — one that retrieves data — within the maximum duration window after the policy was installed or modified.

    For example, suppose you set the duration window in a metric-absence policy to 30 minutes. The condition isn't met if the subsystem that writes metric data has never written a data point. The subsystem needs to output at least one data point and then fail to output additional data points for 30 minutes.

    Meaning, you will need at least 1 datapoint (insert job) to have an incident created for the metric to be missing.

    There are two options:

    1. Create an artificial log entry to get the metric started and have at least one time series and data point.
    2. Run an insert job that would match the log-based metric that was created to get the metric started.

    With regards to your last question, the metric you created is not active because there hasn't been any written data points to it within the previous 24 hours. As mentioned above, the metric must have at least 1 datapoint written to it.

    Refer to custom metrics quota for more info.