Search code examples
ruby-on-railskubernetesopen-telemetryexporter

Unable to export traces to OpenTelemetry Collector on Kubernetes

I am using the opentelemetry-ruby otlp exporter for auto instrumentation: https://github.com/open-telemetry/opentelemetry-ruby/tree/main/exporter/otlp

The otel collector was installed as a daemonset: https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-collector

I am trying to get the OpenTelemetry collector to collect traces from the Rails application. Both are running in the same cluster, but in different namespaces.

We have enabled auto-instrumentation in the app, but the rails logs are currently showing these errors:

E, [2022-04-05T22:37:47.838197 #6] ERROR -- : OpenTelemetry error: Unable to export 499 spans

I set the following env variables within the app:

OTEL_LOG_LEVEL=debug
OTEL_EXPORTER_OTLP_ENDPOINT=http://0.0.0.0:4318

I can't confirm that the application can communicate with the collector pods on this port. Curling this address from the rails/ruby app returns "Connection Refused". However I am able to curl http://<OTEL_POD_IP>:4318 which returns 404 page not found.

From inside a pod:

# curl http://localhost:4318/
curl: (7) Failed to connect to localhost port 4318: Connection refused

# curl http://10.1.0.66:4318/
404 page not found

This helm chart created a daemonset but there is no service running. Is there some setting I need to enable to get this to work?

I confirmed that otel-collector is running on every node in the cluster and the daemonset has HostPort set to 4318.

Solution

  • The problem is with this setting:

    OTEL_EXPORTER_OTLP_ENDPOINT=http://0.0.0.0:4318

    Imagine your pod as a stripped out host itself. Localhost or 0.0.0.0 of your pod, and you don't have a collector deployed in your pod.

    You need to use the address from your collector. I've checked the examples available at the shared repo and for agent-and-standalone and standalone-only you also have a k8s resource of type Service.

    With that you can use the full service name (with namespace) to configure your environment variable.
    Also, the Environment variable now is called OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, so you will need something like this:

    OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=<service-name>.<namespace>.svc.cluster.local:<service-port>