I've got a dataflow job that pulls messages off of several Google Pub/Sub topics, does some parallel processing on the individual elements contained within those messages, then passes the collection on for further consumption by various resources. I'd like to put together a Stackdriver dashboard showing how many individual elements have been processed for each topic. Each ParDo step outputs a PCollection.
I've set up a dashboard using ElementCount, but I'm only able to filter by job, not by step. If I mouseover the line in the chart produced using ElementCount, I can see counts for every single step. Indeed, it does appear that the metrics for these are being reported, as I can use the gcloud commandline utility in the following manner:
gcloud beta dataflow metrics list [jobid] --filter ElementCount
...
name:
context:
original_name: extract_value_topic_1/Map-out0-ElementCount
output_user_name: extract_value_topic_1/Map-out0
name: ElementCount
origin: dataflow/v1b3
scalar: 7000
updateTime: '2017-05-03T18:13:22.804Z'
---
name:
context:
original_name: extract_value_topic_2/Map-out0-ElementCount
output_user_name: extract_value_topic_2/Map-out0
name: ElementCount
origin: dataflow/v1b3
scalar: 12000
updateTime: '2017-05-03T18:13:22.804Z'
I have several of these, but I don't see a straightforward way of building Stackdriver charts based on them (aside from logging to the console for every element processed then using that to generate a log-based metric, but that seems like it'd be extremely inefficient on a number of levels.) Am I missing something? How would one create a chart based on these ElementCounts?
Edit: Additionally, if I open up the Metrics Explorer I can enter dataflow/job/element_count into the search box then pcollection into the filter box, but I'm unable to build a dashboard with this chart in it as the filter selection in the dashboard chart builder does not allow for filtering by pcollection.
Unfortunately, you currently cannot build a dashboard with a filter on a metric label. As you noticed, the new (Beta) Metric Explorer provides the filtering functionality and the Stackdriver team is actively working on providing that functionality to the dashboard charts as well. I will follow up if I receive any further updates or details from the Stackdriver team. --Andrea