SCDF Metrics Collector - Include Prometheus metrics

Question

I am using SCDF with Spring Boot 2.x metrics and SCDF metrics collector to collect metrics from my Spring Boot app. I really do not uderstand the logic of the collector regarding the aggregateMetricsdata.

When I am fetching the list of metrics collected for my stream, I only have the one starting with integration.channel.*and thus I have on ly the mean value. I tried everything to see other metrics appearing like the one exposed by the /actuator/prometheus endpoint.

I think I have misunderstand the way the metrics are aggregated. I noticed that SCDF add automatically some properties to metrics and I would like to apply these properties to all my metrics exposed in order to collect them all.

{
  "_embedded": {
    "streamMetricsList": [
      {
        "name": "poc-stream",
        "applications": [
          {
            "name": "poc-message-sink",
            "instances": [
              {
                "guid": "poc-stream-poc-message-sink-v7-75b8f4dcff-29fst",
                "key": "poc-stream.poc-message-sink.poc-stream-poc-message-sink-v7-75b8f4dcff-29fst",
                "properties": {
                  "spring.cloud.dataflow.stream.app.label": "poc-message-sink",
                  "spring.application.name": "poc-message-sink",
                  "spring.cloud.dataflow.stream.name": "poc-stream",
                  "spring.cloud.dataflow.stream.app.type": "sink",
                  "spring.cloud.application.guid": "poc-stream-poc-message-sink-v7-75b8f4dcff-29fst",
                  "spring.cloud.application.group": "poc-stream",
                  "spring.cloud.dataflow.stream.metrics.version": "2.0"
                },
                "metrics": [
                  {
                    "name": "integration.channel.input.send.mean",
                    "value": 0,
                    "timestamp": "2018-10-25T16:34:39.889Z"
                  }
                ]
              }
            ],
            "aggregateMetrics": [
              {
                "name": "integration.channel.input.send.mean",
                "value": 0,
                "timestamp": "2018-10-25T16:34:52.894Z"
              }
            ]
          },
...

I have some Micrometer counters that I want to get the values with the Metrics collector. I know they are well exposed because I have set all the properties right and I even went into the Docker container launched to check the endpoints.

I have read that

When deploying applications, Data Flow sets the spring.cloud.stream.metrics.properties property, as shown in the following example:

spring.cloud.stream.metrics.properties=spring.application.name,spring.application.index,spring.cloud.application.*,spring.cloud.dataflow.*

The values of these keys are used as the tags to perform aggregation. In the case of 2.x applications, these key-values map directly onto tags in the Micrometer library. The property spring.cloud.application.guid can be used to track back to the specific application instance that generated the metric.

Does that mean that I need to specifically add these properties myself into the tags of all my metrics ? I know I can do that by having a Bean MeterRegistryCustomizerreturning the following : registry -> registry.config().commonTags(tags) with tags beeing the properties that SCDF normally sets itself for integrationmetrics. Or SCDF adds to ALL metrics the properties ?

Thanks !

Answer 1

while your observation about the MetricsCollector is "generally" correct, I believe there is an alternative (and perhaps cleaner) way to achieve what you've been trying by using the SCDF Micrometer metrics collection approach. I will try to explain both approaches below.

As the MetricsCollector precedes in time the Micrometer framework they both implement a quite different metrics processing flows. The primary goal for the Metrics Collector 2.x was to ensure backward compatibility with SpringBoot 1.x metrics. The MetricsCollector 2.x allows mixing metrics coming from both SpringBoot 1.x (pre micrometer) and Spring Boot 2.x (e.g. micrometer) app starters. The consequence of this decision is that the Collector 2.x supports only the common denominator of metrics available in Boot 1.x and 2.x. This requirement is enforced by pre-filtering only the integration.channel.* metrics. At the moment you would not be able to add more metrics without modifying the metrics collector code. If you think that supporting different Micrometer metrics is more important than having backward compatibility with Boot 1.x then please open an new issue in Metrics Collector project. Still I believe that the approach explained below is better suited for you case!

Unlike the MetricsCollector approach, the "pure" Micrometer metrics are send directly to the selected Metrics registry (such as Prometheus, InfluxDB, Atlas and so on). As illustrated in the sample, the collected metrics can be analyzed and visualized with tools such as Grafana. Follow the SCDF Metrics samples to setup your metrics collection via InfluxDB (or Prometheus) and Grafana. Later would allow you explore any out-of-the-box or custom Micrometer metrics. The downside of this approach (for the moment) is that you will not be able to visualize those metrics in the SCDF UI's pipeline. Still if you find it important to have such visualization inside the SCDF UI please open a new issue in the SCDF project (I have WIP for the Altals Micrometer Registry).

I hope that this sheds some light on the alternative approaches. We would be very interested to hear your feedback.

Cheers!