Custom OpenMetrics Not Being Propagated to DataDog

Question

I am using the prometheus-fastapi-instrumentator package to expose my custom metrics but they don't seem to be picked up by DataDog.

I'm experiencing a lot of trouble getting DataDog to scrape my Counter metrics. Additionally, Histogram buckets don't seem to be going through as distribution metrics.

Does anyone have any clue as to what the issue could be?

Here is my monitoring.py file: https://github.com/rileyhun/fastapi-ml-example/blob/main/app/core/monitoring.py

Reproducible Example:

git clone https://github.com/rileyhun/fastapi-ml-example.git

docker build -t ${IMAGE_NAME}:${IMAGE_TAG} -f Dockerfile .
docker tag ${IMAGE_NAME}:${IMAGE_TAG} rhun/${IMAGE_NAME}:${IMAGE_TAG}
docker push rhun/${IMAGE_NAME}:${IMAGE_TAG}

minikube start --driver=docker --memory 4g --nodes 2
kubectl create namespace monitoring
helm install prometheus-stack prometheus-community/kube-prometheus-stack -n monitoring

kubectl apply -f deployment/wine-model-local.yaml
kubectl port-forward svc/wine-model-service 8080:80

python api_call.py

Answer 1

Is the datadog-agent configured to pull your metrics or are you pushing metrics to dogstatsd?

If the datadog-agent is pulling, make sure you follow the instructions here https://docs.datadoghq.com/integrations/guide/prometheus-host-collection/:

The instructions above have more detail, but what you're generally doing is:

1. Make sure your server returns the prometheus metrics at an endpoint. You may want to expose this using a different port that is kept internal.
2. Enable the openmetrics integration by adding the config to the agent so it knows that it needs to pull prometheus metrics from the endpoint you exposed in the above step. If you have a kubernetes setup then this configuration specified on the application end, usually through the Deployment resource in spec.template.metadata.annotations which would look something like:

        ad.datadoghq.com/{name of container declared in spec.containers.name}.check_names : '["openmetrics"]'
        ad.datadoghq.com/{name of container declared in spec.containers.name}.init_configs : '[{}]'
        ad.datadoghq.com/{name of container declared in spec.containers.name}.instances : |
          [
            {
            "prometheus_url" : "http://%%host%%:%%port_0%%/metrics",
            "namespace" : "",
            "metrics": ["*"],
            "tags": {"service": "{name of service for datadog}"},
            "send_histograms_buckets": true,
            "send_distribution_buckets": true,
            "send_distribution_counts_as_monotonic": true
            }
          ]

3. Restart datadog-agent or the deployment if using kubernetes