We're using the OpenTelemetry collector agent to send logs and traces to Elastic Cloud. We now want to incorporate metrics as well.
We have a mix of bespoke apps in addition to some FOSS (Free Open Source Software), COTS (Commercial Off The Shelf) packages, and SaaS (Software as a Service) services. This is pretty typical for most organizations.
Some of the apps/services already expose (or can expose) metrics in Prometheus format e.g. PostgreSQL, RabbitMQ, AWS CloudWatch metrics, etc. There's an OpenTelemetry Prometheus receiver that can scape metrics (similar to Prometheus Server), but the problem I can forsee is that Elastic Cloud (which is our back-end observability platform) expects metrics to be sent in OTLP/proto format. For bespoke apps, we can switch to pushing metrics (in OTLP/proto format) to the OpenTelemetry collector agent, so the bespoke apps are probably fine; but for FOSS/COTS/SaaS that provide Prometheus endpoints, how could we get this to work? Is there a way to convert the text-based Prometheus metrics that are scraped into OTLP/proto format in the OpenTelemetry collector agent? Ideally we don't need to introduce a different agent, and we definitely don't want a separate back-end observability platform e.g. Prometheus/Grafana for metrics - we want ALL telemetry to go Elastic.
Cheers
For the immediate question, you are able to scrape metrics via the collector and send them via OTLP to any OTLP compatible backend (elastic included!) Depending on how many metrics you need to scrape, there are different methods for deploying the otel collector. If you’re in Kubernetes already, there are helm charts and an operator available to make it easy.