Search code examples
prometheusmetricsopen-telemetry

Should request duration metrics be in seconds or milliseconds? (Potential discrepancy between Prometheus and OpenTelemetry)


There seems to be a discrepancy between the OpenTelemetry and Prometheus standards about whether to use milliseconds vs seconds for HTTP request duration. Does anyone know if this is getting resolved, or which standard a project should support?

Here is the OpenTelemetry standard requiring milliseconds for HTTP duration: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/http-metrics.md#metric-httpclientduration

While Prometheus recommends seconds: https://prometheus.io/docs/practices/naming/#base-units

Here's an example of a project (Pyrra) that is choosing to only support seconds, citing the Prometheus guidelines: https://github.com/pyrra-dev/pyrra/issues/667#issuecomment-1484950608

And here's an example of another project (Istio) which chooses to only support milliseconds and cites the OTel standard: https://github.com/istio/istio/issues/44168#issuecomment-1491224740 https://istio.io/latest/docs/reference/config/metrics/#metrics

Which standard should a project support? Is it "incorrect" for Pyrra to favor the Prometheus recommendation over the OpenTelemetry standard? Maybe there's an OpenTelemetry SIG that I should reach out to about this issue? Any insight is appreciated!

As a user of both projects, I'm working around this by adding a recording rule to map Istio's milliseconds-based metric to a seconds-based metric that Pyrra expects. But I wonder if there's an opportunity to fix this upstream...


Solution

  • OpenTelemetry has recently decided to align duration metric units with the prometheus convention of seconds. We're still working out how exactly this change will be rolled out, given that so many instrumentations already record durations in milliseconds. Please follow this issue details and updates.