Search code examples
grafanadashboardopen-telemetryopen-telemetry-collectortempo

How to create a Grafana Dashboard for my OpenTelemetry instruments > Collector > Tempo setup

I am new to all this architecture, and I am a bit lost.

I have integrated all these instruments in my Rails application:

gem "opentelemetry-exporter-otlp"
gem "opentelemetry-instrumentation-mysql2"
gem "opentelemetry-instrumentation-net_http"
gem "opentelemetry-instrumentation-rack"
gem "opentelemetry-instrumentation-rails"

I am sending the metrics to an instance of OpenTelemetryCollector, and from there to a Tempo instance.

In Grafana, I have configured the Tempo datasource.

I can see the Traces in the Grafana > Explorer:

enter image description here

But I don't know how to build a proper dashboard with it. I have tried to import a dashboard like this. But the data source is Prometheus, not Loki. The are only 2 dashboards for Tempo on this database. And they don't look like the one I want.

I would like to see the requests, the statuses, the request_time, ... as in the OpenTelemetry APM Dashboard

Any help?

If my architecture is wrong, I am also willing to change it.

Solution

  • The solution came from one of the Grafana community members:

    See doc for OpenTelemetry APM Dashboard

    1. OpenTelemetry Collector: Receive traces and process them to metrics with Span Metrics Connector
    2. Prometheus: Scrape metrics from OpenTelemetry Collector

    You are missing these steps (older collector versions have had spanmetricsprocessor, not connector) - this connector (processor) aggregates Request, Error and Duration (R.E.D) metrics from span data - so you will generate METRICS from TRACES and you will save these generated METRICS into metric storage - Prometheus.