Stackdriver Logs-Based Metrics - need sum over alignment period

Question

We have some stackdriver log entries that look something like this:

{
  insertId:  "xyz"  
  jsonPayload: {
    countOfApples:  100   
    // other stuff
    }
  // other stuff
}

We would like to be able to set up a log-based metric that tells us the total number of apples seen in the past 10 mins (or any alignment period) but I have, thus far, been unable to find a means of doing so despite reading through the documentation.

Attempt 1:

Filter for those log-entries where countOfApples is specified and create a Counter metric with countOfApples as a label.

having done this, I can filter based on the countOfApples being above or below a certain value. I cannot see a means of aggregating based on this value. All the aggregation options seem to apply to the number of log entries matching the filter over the alignment period

Attempt 2:

Filter for those log-entries where countOfApples is specified and create a distribution metric with the Field Name set to jsonPayload.CountOfApples

This seems to get closer because I can now see the apple count in the metrics explorer but I cannot find the correct combination of Aligner/Reducers to just give me the total number of apples over the period? Selecting Aligner:delta & Reducer:sum results in an error message:

This aggregation does not produce a valid data type for a Line plot type. Click here to switch the aligner to sum and the reducer to 99th percentile

Is it possible to just monitor the total sum of all these values over each alignment period?

Answer 1

As of 2019/05/03, it is not possible to create a counter metric based on the values stored in the logs. Putting the values into a label simply exposes them as strings, which lets you filter but not perform aggregations based on those values. According to the documentation, a counter metric counts log entries, not the values in those log entries. As you've noticed, there aren't enough operations available on distribution metrics to do what you want.

For now, your best bet is to write your own custom metric based on those log values. You can do this by exporting your logs to Cloud Pub/Sub and writing some code to process the logs from Pub/Sub and send custom metrics. Alternatively, you could try to configure the Stackdriver monitoring agent to extract the values using the tail plugin, and send them as custom metrics.

If you just need to graph and explore the values (rather than, e.g., use them for alerting), you could try Cloud Datalab.