Search code examples
google-app-enginestackdrivergoogle-cloud-stackdriver

Plotting the value of a user define metric's label in Stackdriver

I'm outputting the following log statement:

[p~my-proect-name/dev.12345].<stdout>: 08:09:28.870 [RequestE671C95C] INFO  - Finished processing 27653 rows

I want to have a line graph showing the total number of rows processed for each day in Stackdriver.

I tried creating a user defined metric in the GCP interface as follows:

enter image description here

But this is what I see in Stackdriver:

enter image description here

As you can see, instead of plotting the actual value of the label (number of processed rows), Stackdriver is just plotting the number of times the label appears (i.e. the number of log statements).

Is it possible to achieve this in Stackdriver?

Solution

  • Instead of capturing the value as a user-defined label, you will need to capture the values as a distribution logs-based metric. You can learn more about distribution metrics in https://cloud.google.com/logging/docs/logs-based-metrics/distribution-metrics.

    The statistics of the values captured from logs are recorded as a Distribution. You can aggregate the distribution values over 1 day to calculate the sum of the values. This is not directly supported in charts today, but you can use the ListTimeSeries API to get the value you want. For example, the query would look like:

    export ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
curl -H "Authorization: Bearer ${ACCESS_TOKEN}" 'https://monitoring.googleapis.com/v3/projects/[PROJECT_ID]/timeSeries?aggregation.alignmentPeriod=86400s&aggregation.crossSeriesReducer=REDUCE_SUM&aggregation.perSeriesAligner=ALIGN_SUM&filter=metric.type="logging.googleapis.com/user/ropo-metrics"&interval.endTime=2018-01-22T00:00:00Z'

    You can also try the API using the Google API Explorer in https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.timeSeries/list

    In the resulting response, you will see a single distribution value for each day and the mean * count is the sum value over that day.

    It is currently not possible to do these advanced query operations in Stackdriver charts today, but such a feature will be supported soon (Disclaimer: I am an engineer working in Stackdriver).