Stackdriver Alerting Policy - Need To Filter On JSON Payload

Question

I have a Python script set up under App Engine that runs through all of the projects within our organisation that collects data on:

• The disks not associated with snapshot policies
• The snapshot policies that do not have "daily" or "weekly" in their name (and are thus outside our naming convention)

Taking the former as the example, the JSON payload within the Stackdriver log is like:

jsonPayload: {
  DiskWithoutPolicy: "True"   
  diskId: "1234567891234567891"   
  diskName: "server-disk3"   
  project: "projectID"   
}

When I go to create the Alerting Policy, Metrics Explorer only allows me to filter on the log's resource labels (listed here: https://cloud.google.com/monitoring/api/resources):

resource: {
  labels: {
   module_id: "get_googlecloud_snapshotstatus"    
   project_id: "projectID"    
   version_id: ""    
   zone: "europe-west1-d"    
  }

Can Metrics Explorer filter on the JSON payload, so I can therefore see the 'diskId', 'diskName' and 'project'?

If not, is there any other way of achieving what I need?

Answer 1

You can create your own user-defined metrics in Stackdriver Logging. This way, you can capture all logs matching a particular filter and expose the fields that you want as metric labels.

For example, I navigate to Stackdriver Logging -> Logs-based Metrics -> Create Metric and select a filter for a GAE application:

resource.type="gae_app" 
logName=("projects/REDACTED/logs/appengine.googleapis.com%2Fstdout" 
OR "projects/REDACTED/logs/appengine.googleapis.com%2Fstderr" 
OR "projects/REDACTED/logs/appengine.googleapis.com%2Fnginx.request" 
OR "projects/REDACTED/logs/appengine.googleapis.com%2Frequest_log") 
resource.labels.module_id="image-demo"
httpRequest.requestMethod="GET"

Requests contain a generic jsonPayload such as:

jsonPayload: {
  appLatencySeconds: "0.000"   
  latencySeconds: "0.001"   
  trace: "4ff777572199f23f4fc97388e75c0acc"   
 }

On the metric editor (right panel) under Labels there is a Field name dropdown selector that includes our jsonPayload fields:

In our case we select jsonPayload.trace and now we can filter our custom metric by trace label in the Metrics Explorer:

Note that you can create a Stackdriver Monitoring alert directly from the list of user-defined metrics (Create alert from metric):